Information processing device, storage medium storing information processing program, information processing system, and information processing method

ABSTRACT

An example information processing system includes a camera. The information processing system calculates, based on a camera image obtained by the camera, a position or a direction of an object (e.g., a hand of a user) included in the camera image. The information processing system instructs an output device to produce an output in accordance with the position or the direction of the object. The information processing system displays, on a display device, a guide image including a range image representing a range and an indicator image whose position or direction changes in accordance with a change in the output within the range.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2015-24010, filed onFeb. 10, 2015, is herein incorporated by reference.

FIELD

The present technique relates to an information processing device, astorage medium storing an information processing program, an informationprocessing system, and an information processing method for performingprocesses based on a captured image obtained by an image-capturingdevice.

BACKGROUND AND SUMMARY

There are conventional portable game devices including a display and acamera (image-capturing device) provided on the back side of thedisplay. These game devices obtain a captured image in response to ashutter operation by the user to analyze a characteristic portion of thecaptured image obtained and to display, on the display, an imageobtained by superimposing a virtual character over the captured imagebased on the analysis results.

Where captured images used as an input, there is a demand for improvingthe controllability.

Therefore, the present application discloses an information processingdevice, a storage medium storing an information processing program, aninformation processing system, and an information processing method,with which it is possible to improve the controllability.

(1)

An example information processing device described herein includes animage-capturing device, a position calculation unit, an output controlunit and a guide display unit. The position calculation unit calculates,based on a captured image obtained by the image-capturing device, aposition or a direction of an object included in the captured image. Theoutput control unit instructs an output device to produce an output inaccordance with the position or the direction of the object. The guidedisplay unit displays, on a display device, a guide image including arange image representing a range and an indicator image whose positionor direction changes in accordance with a change in the output withinthe range.

(2)

The image-capturing device may capture an image in a side surfacedirection of a housing of the information processing system.

(3)

The indicator image may be an image representing the object.

(4)

The guide display unit may display, on the display device, a guide imageincluding a predetermined device reference image so that a positionalrelationship between the indicator image and the predetermined devicereference image corresponds to a positional relationship between theobject and the image-capturing device.

(5)

When a position of the object cannot be calculated, the guide displayunit may change a display mode of the guide image, as compared with thatwhen the position of the object is calculated.

(6)

When the position of the object calculated by the position calculationunit is further away toward an un-calculatable area from a predeterminedreference position by a predetermined distance or more, the guidedisplay unit may change a display mode of the guide image, as comparedwith that when the calculated position is within the predetermineddistance.

(7)

The position calculation unit may calculate, based on the capturedimage, a position of the object with respect to an image-capturingdirection of the image-capturing device.

(8)

The output control unit may control the output so that the output is inaccordance with a relationship between the position of the objectcalculated by the position calculation unit and a predeterminedreference position. The guide display unit may display, on the displaydevice, a guide image including a process reference image placed at aposition corresponding to the reference position within the rangerepresented by the range image.

(9)

Another example information processing device described herein includesa detection unit, a calculation unit, an output control unit and a guidedisplay unit. The detection unit detects an object present at a positionaway from the information processing device. The calculation unitcalculates the position or the direction of the object based on thedetection result of the detection unit. The output control unitinstructs an output device to produce an output in accordance with theposition or the direction of the object. The guide display unitdisplays, on a display device, a guide image including a range imagerepresenting a range and an indicator image whose position or directionchanges in accordance with a change in the output within the range.

The present specification also discloses a storage medium storing aninformation processing program instructing a computer of an informationprocessing system to function as the various units of the informationprocessing device set forth above. Note that the computer of theinformation processing system may be a computer of informationprocessing device, or a computer of another information processingdevice separate from the information processing device. The presentspecification also discloses an information processing system includingthe various units of the information processing device set forth above.The present specification also discloses an information processingmethod to be performed on the information processing system set forthabove.

With the information processing device, the storage medium storing aninformation processing program, the information processing system, andthe information processing method, it is possible to improve thecontrollability.

These and other objects, features, aspects and advantages will becomemore apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front view showing a non-limiting example portable device ofthe present embodiment;

FIG. 2 is a right side view showing a non-limiting example portabledevice;

FIG. 3 is a back view showing a non-limiting example portable device;

FIG. 4 is a block diagram showing an internal configuration of anon-limiting example portable device;

FIG. 5 shows how a non-limiting example portable device is used;

FIG. 6 shows a non-limiting example captured image;

FIG. 7 shows an example of a captured image where the hand position hasmoved from the state shown in FIG. 6;

FIG. 8 shows another example of a captured image where the hand positionhas moved from the state shown in FIG. 6;

FIG. 9 shows an example of a captured image where the hand is in an openshape;

FIG. 10 shows an example of a captured image where the hand is in aclosed shape;

FIG. 11 is a flow chart showing an example of the flow of a processperformed by a non-limiting example portable device;

FIG. 12 shows an example of how a user performs a fanning action;

FIG. 13 shows an example of how a user performs a rotating action;

FIG. 14 shows an example of how a user performs a flicking action;

FIG. 15 shows an example of an area of a hand detected from a capturedimage;

FIG. 16 shows an area of a hand extracted from a captured image wherethe hand is in a closed shape;

FIG. 17 shows an example of an operation performed in a secondembodiment;

FIG. 18 shows an example where an object in a virtual space is moved byusing a cursor;

FIG. 19 is a flow chart showing an example of the flow of a processperformed by a non-limiting example portable device of the secondembodiment;

FIG. 20 shows an example of how a user performs a direction changingoperation according to a variation of the second embodiment;

FIG. 21 shows an example of an area of a hand detected from a capturedimage for two cases where the hand is oriented differently;

FIG. 22 shows an example of an operation performed in a thirdembodiment;

FIG. 23 shows an example of the relationship between the hand positionand the information process;

FIG. 24 shows an example of a guide image;

FIG. 25 shows an example of a guide image to be displayed when the handposition is in the prior warning range;

FIG. 26 shows an example of a guide image to be displayed when the handposition is un-calculatable;

FIG. 27 is a flow chart showing an example of the flow of a processperformed by a non-limiting example portable device of the thirdembodiment;

FIG. 28 shows an example of a variation of the guide image;

FIG. 29 shows another example of a variation of the guide image;

FIG. 30 shows an example of how a user performs a touch operation on aportable device;

FIG. 31 shows an example of the transition through captured images thatare obtained during a touch operation;

FIG. 32 shows an example of the flow of a game to be played by using atouch operation;

FIG. 33 is a flow chart showing an example of the flow of a processperformed by a non-limiting example portable device of a fourthembodiment; and

FIG. 34 shows an example of an operation using a hand.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS FirstEmbodiment

An information processing device, an information processing system, aninformation process program and an information processing method of thepresent embodiment will now be described. An information processingdevice of the first embodiment captures an image of an object, such as ahand of a user, by means of a camera (an infrared camera) so as todetermine the action of the user based on the captured image.

[1. Configuration of Portable Device]

First, referring to FIG. 1 to FIG. 5, an example of a configuration of aportable device, which is an example of the information processingdevice, will be described. FIG. 1 is a front view showing an example ofa portable device of the first embodiment. FIG. 2 is a right side viewshowing an example of a portable device. FIG. 3 is a back view showingan example of a portable device. In the first embodiment, a portabledevice 1 is a portable information processing device that can be held ina hand and operated by a user. For example, the portable device 1 may bea hand-held device such as a portable game device, a portable telephone,a smartphone, a tablet terminal or a camera, or may be a terminal thatcan be worn by a user such as a wristwatch-shaped terminal.

As shown in FIG. 1, the portable device 1 includes a display 2, a touchpanel 3, an infrared camera 4, a distance measuring sensor 5, inputbuttons 6 (6A to 6D), an illuminating section 7 and a projector 8, whichare accommodated in a housing 10. The housing 10 (the portable device 1)has a plate-like shape, and is sized so that it can be held by a userwith one hand or both hands. The housing 10 has an elongate shape in alandscape position.

For example, the display 2 may be a liquid crystal display device, anorganic EL display device, or the like, or may by any other suitabledisplay device. The screen of the display 2 is provided so as to beexposed on the front surface (T5 surface) of the housing 10. The touchpanel 3 is provided on the screen of the display 2 for detecting theposition at which the screen has been touched by a user. The touch panel3 may be a touch panel capable of one-point detection or a touch panelcapable of multi-point detection, and may be of any suitable type suchas capacitive or resistive, for example.

The input buttons 6A to 6D each receive an input (pressing of a button)by a user. The input buttons 6A to 6D are each provided at such aposition that it can be reached by a finger of a user when the userholds the opposing sides of the portable device 1. Specifically, theinput buttons 6A and 6C are located so that they can be reached byfingers of the right hand when the user holds the portable device 1 withthe right hand, wherein the input button 6A is provided at such aposition that it can be reached by the thumb of the right hand, and theinput button 6C is provided at such a position that it can be reached bythe index finger or the middle finger of the right hand. The inputbuttons 6B and 6D are located so that they can be reached by fingers ofthe left hand when the user holds the portable device 1 with the lefthand, wherein the input button 6B is provided at such a position that itcan be reached by the thumb of the left hand, and the input button 6D isprovided at such a position that it can be reached by the index fingeror the middle finger of the left hand. As shown in FIG. 1, the inputbuttons 6A and 6B are provided on the front surface (T5 surface) of thehousing 10, and the input buttons 6C and 6D are provided on the topsurface (T4 surface) of the housing 10. Note that input sections forreceiving inputs from a user may include, in addition to the inputbuttons 6A to 6D, a cross-shaped key, an analog stick, or the like, fordirection inputs.

The infrared camera 4 includes a lens, and a sensor capable of sensinglight (infrared rays; specifically, near infrared rays). The sensor ofthe infrared camera 4 is an image sensor in which elements capable ofsensing infrared rays are arranged in a matrix, and the elements of theimage sensor receive and convert infrared rays into electric signals,outputting a two-dimensional infrared image.

The distance measuring sensor 5 emits light (e.g., infrared light) froma light source provided in the distance measuring sensor 5 and receivesthe light having been reflected off an object by means of alight-receiving element, thereby measuring the distance to the object.The distance measuring sensor 5 may be of any type, including atriangulation type and a TOF (Time Of Flight) type. The light source ofthe distance measuring sensor 5 may be an LED, a laser diode, or thelike, emitting infrared light in a particular direction.

The illuminating section 7 outputs infrared rays at a predetermined timeinterval (e.g., an interval of 1/60 sec). The illuminating section 7outputs infrared rays in sync with the timing with which the infraredcamera 4 captures images. The illuminating section 7 outputs infraredrays onto a predetermined range in the right side surface direction ofthe portable device 1. Infrared rays output by the illuminating section7 are reflected by an object, and the reflected infrared rays arereceived by the infrared camera 4, thereby obtaining an image of theinfrared rays. Note that the illuminating section 7 may be used forcapturing an infrared image by the infrared camera 4 and for measuringthe distance by the distance measuring sensor 5. That is, the infraredlight from the illuminating section 7 may be used both for capturing animage by means of the infrared camera 4 and for measuring the distanceby means of the distance measuring sensor 5.

The projector 8 includes a light source for emitting visible light, andprojects text, an image, etc., onto a projection plane (a screen or thehand of a user as will be described later) by using light from the lightsource.

The infrared camera 4, the distance measuring sensor 5, the illuminatingsection 7 and the projector 8 are provided on a side surface of thehousing 10 (e.g., the right side surface: T1 surface). Specifically, theimage-capturing direction (optical axis) of the infrared camera 4 isoriented vertical to the right side surface. The detecting direction ofthe distance measuring sensor 5 and the direction in which the projector8 outputs light are also vertical to the right side surface. That is,when a user holds the portable device 1 with the left hand, the infraredcamera 4 captures an image of the space in the right side surfacedirection of the portable device 1, and the distance measuring sensor 5measures the distance to an object existing in the space in the rightside surface direction of the portable device 1. The projector 8projects an image, or the like, by outputting visible light in the samedirection as the infrared camera 4 and the distance measuring sensor 5.

An outside camera 9 is provided on the back surface (T6 surface) of theportable device 1 (FIG. 3). The outside camera 9 is typically capable ofcapturing an image in the direction vertical to the image-capturingdirection of the infrared camera 4, and is capable of capturing an imagein the direction vertical to the back surface. The outside camera 9includes a lens, and an image sensor capable of sensing visible light.The outside camera 9 captures an image of the space in the back surfacedirection, obtaining a color image (RGB image). Note that a camera maybe provided on the front surface in addition to the outside camera 9 onthe back surface, or a camera may be provided on the front surface (thesurface on which the screen of the display 2 is provided) withoutproviding the outside camera 9 on the back surface.

FIG. 4 is a block diagram showing an example of an internalconfiguration of the portable device 1. As shown in FIG. 4, in additionto the various sections mentioned above, the portable device 1 includesa vibrator 11, a microphone 12, a speaker 13, a control section 14, acommunication section 15, an attitude detection section 16, a GPSreceiver 17 and a geomagnetic sensor 18. The portable device 1 alsoincludes a battery (not shown), and receives power supply from thebattery. These sections are accommodated in the housing 10.

The control section 14 is connected to, and controls, various sectionsincluding the display 2, the touch panel 3, the infrared camera 4, thedistance measuring sensor 5, the input button 6, the illuminatingsection 7, the projector 8, the vibrator 11, the microphone 12, thespeaker 13, the communication section 15, the attitude detection section16, the GPS receiver 17 and the geomagnetic sensor 18.

Specifically, the control section 14 includes a CPU, a memory, etc., forperforming a predetermined process based on a predetermined program(e.g., an application program for performing game processes, imageprocesses, and various arithmetic operations) stored in a storage device(not shown) (e.g., a non-volatile memory or a hard disk) provided in theportable device 1. For example, the control section 14 may obtain andanalyze an image from the infrared camera 4, calculate the distance toan object based on the signal from the distance measuring sensor 5, orperform a process in accordance with input signals from the touch panel3 and the input button 6. Then, the control section 14 generates animage based on the results of the predetermined process, and outputs theimage on the display 2. Note that the program for performing thepredetermined process may be downloaded from outside via thecommunication section 15.

The vibrator 11 operates based on an instruction from the controlsection 14 and vibrates the entire portable device 1. The vibrator 11 isprovided at a predetermined position such that vibrations are easilytransmitted to the hand of a user (e.g., in the central portion, or at aposition shifted left or right therefrom, inside the housing 10).

The microphone 12 and the speaker 13 are used for inputting/outputtingsounds. The communication section 15 is used for communicating withother devices by a predetermined communication scheme (e.g., wirelessLAN, etc.). The attitude detection section 16 is, for example, anacceleration sensor or an angular velocity sensor for detecting theattitude of the portable device 1.

The GPS receiver 17 receives a signal from a GPS (Global PositioningSystem) satellite, and the portable device 1 can calculate the positionof the portable device 1 itself by receiving the signal. For example,when a predetermined operation is performed at a particular position(e.g., a gesture input or a button input using the infrared camera 4,shaking the portable device 1, etc., to be described later), theportable device 1 may display an object associated with the particularposition. For example, where a game is played on the portable device 1,and the portable device 1 is at a particular position, an objectassociated with the particular position may appear in the game.

The geomagnetic sensor 18 is a sensor capable of detecting the directionand the magnitude of a magnetic field. For example, based on thedetection result of the geomagnetic sensor 18, the portable device 1 maydetermine whether it is facing a particular azimuthal direction, and theportable device 1 may display an object when a predetermined operation(e.g., a gesture input mentioned above) is performed in a particularazimuthal direction. For example, where a game is played on the portabledevice 1, an object associated with a particular azimuthal direction mayappear in the game. The portable device 1 may combine together GPSinformation obtained by the GPS receiver 17 and azimuthal directioninformation obtained by the geomagnetic sensor. For example, if theportable device 1 is at a particular position and is facing a particularazimuthal direction, the portable device 1 may display an objectassociated with the particular position and the particular azimuthaldirection, or such an object may appear in the game.

FIG. 5 shows an example of how the portable device 1 is used. As shownin FIG. 5, a user can hold the portable device 1 with one hand and makegesture inputs using the other hand. Note that a gesture input may beany input made with an operation object that is operated (moved) by auser. An operation object may be the body of the user (which may be thewhole body or a part of the body such as a hand or the face), an objectheld by the user, or may be both the body and an object. The portabledevice 1 may recognize the shape of the operation object as a gestureinput, may recognize the movement of the operation object as a gestureinput, or may recognize a combination of the shape and the movement as agesture input. For example, a user may make a gesture input by using theshape, the movement, the position (with respect to the portable device1), the orientation (attitude), etc., of the hand.

In the first embodiment, the portable device 1 captures an image(infrared image) by means of the infrared camera 4, and determines agesture input based on the captured image. Specifically, when a capturedimage is obtained from the infrared camera 4, the portable device 1calculates the shape, the position, and/or the orientation (attitude) ofthe object based on the captured image obtained. Note that based on aplurality of captured images obtained through repetition, the portabledevice 1 may calculate the movement of the object (the change of shape,the moving direction, the moving speed, etc.). The portable device 1determines a gesture input based on these calculation results.

[2. Outline of Process Performed in Portable Device 1]

Next, a process performed by the portable device 1 in the firstembodiment will be outlined. Referring to FIG. 6 to FIG. 10, a processwill be described below, in which the portable device 1 detects a handof a user by means of an infrared camera and calculates the position andthe shape of the hand.

Note that the first embodiment (and similarly in the second to fourthembodiments to be described below) assumes a case where a user makes agesture input by using a hand, and the portable device 1 detects thehand of the user and calculates the position and the shape thereof. Notehowever that the object to be detected by the portable device 1 may beany suitable object. For example, in other embodiments, the portabledevice 1 may detect a predetermined operation member (e.g., a pen) heldby a user.

FIG. 6 shows an example of a captured image. In the first embodiment, acaptured image (infrared image) captured by the infrared camera 4contains information on the brightness value of each pixel (i.e., avalue representing the brightness of each pixel of a captured image).Now, infrared rays are output from the illuminating section 7 onto anobject (a hand of a user in FIG. 6) located in the image-capturing rangeof the infrared camera 4, and the reflected light is detected by theinfrared camera 4. Therefore, the brightness value is higher in the areaof the object in the captured image. For example, if a user places ahand near the infrared camera 4 (e.g., in a range of 40 cm or less), ahand area 21 is included in the captured image as shown in FIG. 6. Wherethe palm portion is located close to the infrared camera 4 with the armportion being located gradually away from the infrared camera 4, thebrightness value of the arm portion in the hand area 21 gradually lowersin the direction away from the palm portion.

Note that it is assumed in the first embodiment that the brightnessvalue of each pixel in the captured image is represented in multiplesteps (e.g., 256 steps). Note however that in FIG. 6, for ease ofunderstanding, the hatched area represents the portion where thebrightness value is relatively low and the unhatched area (the hand area21) represents the portion where the brightness is relatively high.

In the first embodiment, the portable device 1 calculates the handposition based on the captured image. In the first embodiment, theportable device 1 calculates the three-dimensional position of the hand,i.e., the position in the two-dimensional direction (the x-axisdirection and the y-axis direction shown in the figures) perpendicularto the image-capturing direction of the infrared camera 4, and theposition with respect to the image-capturing direction of the infraredcamera 4 (the z-axis direction shown in the figures).

(Calculating Position with Respect to x-Axis Direction and y-AxisDirection)

The position in the two-dimensional direction perpendicular to theimage-capturing direction of the infrared camera 4 is calculated byusing the center-of-gravity position regarding the brightness value ofthe captured image (the position P1 shown in FIG. 6). Now, thecenter-of-gravity position in the captured image refers to thecenter-of-gravity position of the entire captured image obtained withthe brightness value of each pixel of the captured image being used as aweight. The center-of-gravity position can be obtained by calculating,for each pixel, a two-dimensional value obtained by multiplyingtwo-dimensional coordinates representing the position of the pixel bythe brightness value of the pixel, and then dividing the total sum ofthe calculated two-dimensional values by the total sum of the brightnessvalues of the pixels. Note that if the captured image includes no area,other than the hand area 21, where the brightness value is high and thearea is large (i.e., if no object other than the hand is captured by theinfrared camera 4), the center-of-gravity position will be inside thehand area 21.

FIG. 7 shows an example of a captured image where the hand position hasmoved from the state shown in FIG. 6. As shown in FIG. 7, when the handarea 21 in the captured image moves, the center-of-gravity positionmoves accordingly. Therefore, it can be said that the center-of-gravityposition represents the hand position, and can be used as the handposition. That is, the portable device 1 can detect the hand positionwith respect to the two-dimensional direction by calculating thecenter-of-gravity position.

Note that the portable device 1 is not limited to the configurationwhere the center-of-gravity position itself is used as the handposition, but any position that is determined based on thecenter-of-gravity position may be used as the hand position. Forexample, the hand position may be calculated as a position obtained bymoving the center-of-gravity position in a predetermined movingdirection by a predetermined moving amount. The moving direction and/orthe moving amount may be fixed or may be variable. For example, themoving direction and/or the moving amount may be set variably inaccordance with at least one of the shape of the hand, the hand positionwith respect to the image-capturing direction, and the brightness of theentire captured image. That is, the portable device 1 may determine themethod for calculating the hand position from the center-of-gravityposition in accordance with at least one of the shape of the hand, thehand position with respect to the image-capturing direction, and thebrightness of the entire captured image. Thus, it is possible to moreaccurately calculate the hand position. Note that the method forcalculating the shape of the hand, the hand position with respect to theimage-capturing direction and the brightness of the entire capturedimage will be described later.

As described above, in the first embodiment, the portable device 1calculates the hand position using the center-of-gravity position of thecaptured image without specifying the hand area 21 in the capturedimage. Then, it is possible to calculate the hand position by a simplemethod without the need for an image recognition process for specifyingthe hand area (the outline of the hand). In other words, it is possibleto reduce the process load of the process of calculating the handposition, and it is possible to increase the speed of the process.

Note that the area for which the center-of-gravity position iscalculated does not need to be the area of the entire captured image,but may be any area that is set independently of the hand area 21. Forexample, the portable device 1 may calculate the center-of-gravityposition for an inside area that is at a predetermined distance from theend portion (the perimeter) of the captured image. Then, since the handposition is calculated when the hand area 21 is present in the insidearea, gesture inputs are accepted only when the hand area 21 is presentin the inside area.

(Calculating Position with Respect to z-Axis Direction)

In the first embodiment, the portable device 1 calculates the positionwith respect to the image-capturing direction (the z-axis direction)based on the captured image. The position with respect to theimage-capturing direction (the distance from the infrared camera 4) iscalculated based on the brightness of the (entire) captured image.Specifically, the portable device 1 calculates the average brightnessvalue among pixels of the captured image, and calculates the distancebased on the average value.

FIG. 8 shows another example of a captured image where the hand positionhas moved from the state shown in FIG. 6. The captured image shown inFIG. 8 shows a case where the hand has moved toward the infrared camera4, as compared with the captured image shown in FIG. 6. In such a case,since the proportion of the area of the captured image occupied by thehand area 21 is larger as compared with FIG. 6, the brightness of thecaptured image (the average value described above) is higher than thatof FIG. 6. Thus, the brightness of the entire captured image variesdepending on the distance from the infrared camera 4 to the hand. Thus,the portable device 1 is capable of calculating the hand distance basedon the brightness of the captured image.

The specific method for calculating the distance based on the brightnessof the captured image may be any method. For example, in the firstembodiment, the portable device 1 may store, in advance, informationrepresenting the correlation between the brightness and the distance(which may be a table or may be a function), and calculate the distancebased on the correlation. Note that in other embodiments, the portabledevice 1 may calculate the distance based on the position at a referencepoint in time (e.g., a point in time at which control has started toobtain the captured image) (i.e., a relative position with respect tothe position at the reference point in time) based on the amount ofchange in the brightness of the captured image at the reference point intime.

Note that the portable device 1 can calculate the two-dimensionalposition of the hand from the center-of-gravity position describedabove, irrespective of the hand distance. Note however that thepositional relationship between the center-of-gravity position and thehand area 21 may vary depending on the distance described above.Therefore, the portable device 1 may change the process (method) forspecifying the hand position from the center-of-gravity position inaccordance with the distance described above. For example, where thehand position is calculated as a position obtained by moving thecenter-of-gravity position in a predetermined moving direction by apredetermined moving amount, the portable device 1 may change the movingdirection and/or the moving amount depending on the distance describedabove.

As described above, in the first embodiment, the portable device 1calculates the distance of the hand from the infrared camera 4 by usingthe brightness of the captured image without specifying the hand area 21in the captured image. Then, there is no need for the image recognitionprocess for specifying the hand area 21, and it is possible to calculatethe hand distance with a simple method. In other words, it is possibleto reduce the process load of the process for calculating the handdistance, and it is possible to increase the speed of the process.

Note that the area for which the brightness of the captured image (theaverage value described above) is calculated does not need to be an areaof the entire captured image, but may be any area that is setindependently of the hand area 21. For example, as with the calculationof the center-of-gravity position, the portable device 1 may calculatethe average value described above for an inside area that is at apredetermined distance from the end portion (perimeter) of the capturedimage.

(Specifying Shape of Hand)

In the first embodiment, the portable device 1 calculates the shape ofthe hand based on the captured image. In the first embodiment, theportable device 1 specifies the shape of the hand to be either anopen-hand shape (the open-palm shape) or a closed-hand shape (theclosed-fist shape).

FIG. 9 shows an example of a captured image where the hand is in anopen-hand shape. Note that in FIG. 9, for ease of understanding, thearea of a lower brightness value is not hatched (as opposed to FIG. 6,etc.). In the first embodiment, the portable device 1 sets, on thecaptured image, a determination range 22 for determining the shape ofthe hand, as shown in FIG. 9. The determination range 22 is set based onthe center-of-gravity position P1 described above. Specifically, in thefirst embodiment, the portable device 1 sets the determination range 22as a circular area having a radius of a predetermined length andcentered about the center-of-gravity position P1. Note that the methodfor setting the determination range 22 based on the center-of-gravityposition may be any method. For example, in other embodiments, where aposition different from the center-of-gravity position is calculated asthe hand position, the portable device 1 may set the determination range22 as a circular area centered about the hand position.

There is no limitation on the size and the shape of the determinationrange 22. In the first embodiment, the size of the determination range22 (the diameter of the circular area) is set to be larger than the handarea in a closed position and smaller than the hand area in an openposition. In other words, the position and the size of the determinationrange are set so that the hand area is included inside the determinationrange in the closed-hand position and that portions of the hand area(portions of the fingers) lie outside the determination range in theopen-hand position. Therefore, the size of the determination range 22may be set to vary depending on the distance of the hand from theinfrared camera 4 (in other words, the brightness of the capturedimage).

When the hand is in an open shape, as shown in FIG. 9, finger areas anda wrist (arm) area (the hatched areas shown in FIG. 9) of the hand area21 lie outside the determination range 22. On the other hand, FIG. 10shows an example of a captured image where the hand is in a closedshape. When the hand is in a closed shape as shown in FIG. 10, only thewrist area of the hand area 21 lies outside the determination range 22.Thus, the shape of the hand can be determined based on the relationshipbetween the determination range 22 and the hand area.

Specifically, the portable device 1 determines whether the hand is in anopen shape or a closed shape in accordance with the number of areas ofthe hand area 21 that are lying outside the determination range 22(referred to as “outside areas”). Note that an outside area can bedetected as an area that is lying outside the determination range 22 andhas a relatively high brightness value (e.g., an area where thebrightness value is greater than or equal to a predetermined value). Inthe first embodiment, the portable device 1 determines that the hand isin an open shape when the number of the outside areas is five or more,and determines that the hand is in a closed shape when the number of theoutside areas is one or less. Note that the expression “five or more”(as opposed to “six”) and the expression “one or less” (as opposed to“one”) are used so that the shape of the hand can be determined even ifone of the outside areas (e.g., the wrist area) is not detected due to alow brightness value.

Note that in other embodiments, the portable device 1 may determine theshape of the hand to be one selected from among three or more differentshapes. For example, based on the number of outside areas, the shape ofthe hand can be determined to be the closed-fist shape, the closed-fistshape, or the open-palm shape. Specifically, the portable device 1determines that the hand is in an open shape (the open-palm shape) whenthe number of areas lying outside the determination range 22 is five ormore, determines that the hand is in a shape with two fingers extended(the closed-fist shape) when the number of such areas is two or three,and determines that the hand is in a closed shape (the closed-fistshape) when the number of such areas is one or less.

In other embodiments, the portable device 1 may set a plurality ofdetermination ranges. Then, the portable device 1 may determine theshape of the hand based on the relationship between the determinationranges and the hand area. Specifically, the portable device 1 may settwo determination ranges, which are circular areas of different radiicentered about the hand position. Then, the portable device 1 candistinguish between the shape in which the hand is closed, the shape inwhich the hand is beginning to open, and the shape in which the hand isopen, as the shape of the hand. By setting a plurality of determinationranges as described above, it is possible to specify the shape of thehand more specifically.

As described above, in the first embodiment, the portable device 1determines the shape of the hand by using the determination range 22based on the center-of-gravity position without specifying the hand area21 in the captured image. Then, there is no need for the imagerecognition process for specifying the hand area 21, and it is possibleto specify the shape of the hand with a simple method. In other words,it is possible to reduce the process load of the process for specifyingthe shape of the hand, and it is possible to increase the speed of theprocess.

[3. Specific Example of Process Performed by Portable Device 1]

Next, a specific example of the process performed by the portable device1 will be described. FIG. 11 is a flow chart showing an example of theflow of the process performed by the portable device 1. A series ofprocesses shown in FIG. 11 may be performed under any condition and withany timing. In the first embodiment, the series of processes areperformed while running a predetermined application program (e.g., agame program) that accepts gesture inputs.

Note that in the present application, the process in each step in theflow chart shown in the figure is merely an example, and the order ofsteps may be switched around, or other processes may be performed inaddition to (or instead of) these steps, as long as similar results areobtained. While the present embodiment is described herein assuming thatthe processes of the steps of the flow chart are performed by the CPU ofthe portable device 1, processes of some of the steps of the flow chartmay be performed by a processor or a dedicated circuit other than theCPU. Some of the processes performed by the portable device 1 may beperformed by another information processing device capable ofcommunicating with the portable device 1 (e.g., a server capable ofcommunicating with the portable device 1 via a network). That is, theprocesses shown in FIG. 11 (this also applies to the processes of FIG.19, FIG. 27 and FIG. 33) may be performed through a cooperation of aplurality of information processing devices including the portabledevice 1.

The control section 14 performs the processes shown in FIG. 11 (thisalso applies to the processes of FIG. 19, FIG. 27 and FIG. 33) by usinga memory. That is, the control section 14 stores information (data)obtained through the process steps in a memory, and the control section14 reads out the information from the memory when such information is tobe used in a subsequent process step.

First, in step S1, the control section 14 of the portable device 1obtains, from the infrared camera 4, a captured image captured by theinfrared camera 4. That is, as the infrared camera 4 captures an imageat a predetermined time interval, the control section 14 obtains thecaptured image from the infrared camera 4 and stores data of thecaptured image in a memory.

In step S2, based on the captured image obtained in step S1, the controlsection 14 calculates the two-dimensional position of the hand (theposition with respect to the direction perpendicular to theimage-capturing direction of the infrared camera 4) included in thecaptured image. The two-dimensional position is calculated by the methoddescribed in “(Calculating position with respect to x-axis direction andy-axis direction)” above.

In step S3, based on the captured image obtained in step S1, the controlsection 14 calculates the distance of the hand included in the capturedimage (the position with respect to the image-capturing direction of theinfrared camera 4). The distance is calculated by the method describedin “(Calculating position with respect to the z-axis direction)” above.

In step S4, based on the captured image obtained in step S1, the controlsection 14 specifies the shape of the hand included in the capturedimage (whether the hand is in an open shape or in a closed shape). Theshape of the hand is calculated by the method described in “(Calculatingshape of hand)” above.

In step S5, the control section 14 performs a predetermined processbased on the three-dimensional position and the shape of the handcalculated in the series of processes of steps S2 to S4. Thepredetermined process may be any information process in which thethree-dimensional position and the shape of the hand are used as inputs.For example, the control section 14 performs a process of controlling anobject in a virtual space (e.g., a player character, a cursor, etc.,appearing in the game space) in accordance with the three-dimensionalposition and the shape of the hand. For example, the control section 14may detect a gesture input based on the three-dimensional position andthe shape of the hand, and perform a process in accordance with thedetected gesture input. In the process of step S5, the control section14 may display an image representing the results of the predeterminedprocess on the display 2.

In step S6, the control section 14 determines whether or not to end theseries of processes performed in accordance with the position and theshape of the hand. The specific method of the determination in step S6may be any method. In the first embodiment, if a user gives apredetermined end instruction (e.g., an instruction to end anapplication), it is determined to end the series of processes. If thedetermination result of step S6 is affirmative, the control section 14ends the process shown in FIG. 11. On the other hand, if thedetermination result of step S6 is negative, the control section 14performs the process of step S1 again. Thereafter, the series ofprocesses of steps S1 to S6 are performed repeatedly until it isdetermined in step S6 to end the process.

[4. Function/Effect of First Embodiment]

As described above, in the first embodiment, the portable device 1obtains information regarding the brightness of each pixel based on thecaptured image (referred to also as the “camera image”) (step S1), andcalculates the position information of the captured image representingthe deviation obtained with the brightness of each pixel being used as aweight (the information of the center-of-gravity position) (step S2).Then, the portable device 1 performs a predetermined information processbased on the position information (step S5). Note that while the area(in the captured image) for which the position information is calculatedis the entire area of the captured image in the embodiment describedabove, it may be any area to be set independently of the area of theimage of the object (a hand). That is, “to calculate positioninformation for a captured image” means to include both the embodimentin which the position information is calculated for the entire area ofthe captured image and the embodiment in which the position informationis calculated for a partial area of the captured image. Thepredetermined information process may be any process, and is, forexample, an information process performed in accordance with theposition, the shape and/or the movement of the object (a hand).

As described above, the portable device 1 is capable of calculating theposition of the object by a simple process. The portable device 1 iscapable of calculating the position of the object by a simple methodwithout specifying an area of the object by an image recognitionprocess, etc., for example.

Note that the portable device 1 may calculate information representingthe position of the object based on the position information or maycalculate information representing the change in the position (speed) ofthe object to perform an information process based on the calculatedinformation. Then, it is possible to perform an information process inwhich the position and/or the movement of the object is used as a userinput.

The portable device 1 may calculate information representing the shapeof the object included in the captured image based on the positioninformation (step S4), and perform an information process based on theinformation (step S5). Note that the information process may be aprocess in which the shape of the object is used as an input, or aprocess in which the change in the shape (the movement) of the object isused as an input. As described above, it is possible to perform aninformation process in which the shape of the object and/or the changethereof is used as a user input.

Note that in the embodiment described above, the portable device 1 setsa determination range on a captured image based on the position(center-of-gravity position) on the captured image specified by positioninformation, and calculates the shape of the object (whether the hand isin an open shape or in a closed shape) based on the relationship betweenthe determination range and the area of the image of the object (stepS4). Then, the portable device 1 can specify the shape of the objectwithout specifying the area of the object in the captured image. Theportable device 1 can specify the shape of the object by a simpleprocess since it is not needed to specify the area of the object by animage recognition process, or the like.

Note that the portable device 1 calculates the brightness (the averagebrightness value) of the captured image (step S3), and performs aninformation process based on the position information and the brightness(step S5). Then, as the position information and the brightness are usedas inputs, it is possible to allow for a wider variety of operations.Note that the area for which the brightness of the captured image iscalculated may be the same as, or different from, the area for which theposition information is calculated. The area for which the brightness ofthe captured image is calculated may be set independently of the area ofthe image of the object in the captured image.

The portable device 1 calculates information representing the positionand/or the movement for the image-capturing direction (the z-axisdirection) of the infrared camera 4 (step S3), and performs aninformation process in accordance with the information (step S5). Then,it is possible to calculate the position with respect to theimage-capturing direction by a simple process.

Moreover, the portable device 1 may calculate information representingthe two-dimensional position with respect to the direction (the x-axisdirection and the y-axis direction) generally perpendicular to theimage-capturing direction based on the position information (step S2) toperform the information process based on the information (step S5). Notethat the information may be information representing the two-dimensionalmovement in addition to, or instead of, the two-dimensional position.Then, the portable device 1 can calculate the three-dimensional positionof the object based on the captured image.

In the embodiment described above, the portable device 1 calculates, asthe position information, the center-of-gravity position of the wholearea or a predetermined partial area of the captured image, in which thebrightness of each pixel is used as a weight. Then, the portable device1 can easily calculate an index representing the deviation of brightnessacross the predetermined area where the brightness of each pixel is usedas a weight. Note that in other embodiments, the portable device 1 maycalculate, as the position information, any index representing thedeviation of the area, which is not limited to the center-of-gravityposition.

[5. Variation]

Variations (applications) using the information process of the firstembodiment include the following examples, for example. FIG. 12 shows anexample where a user performs a fanning action. FIG. 13 shows an exampleof how a user performs a rotating action. FIG. 14 shows an example ofhow a user performs a flicking action. The portable device 1 may detectthese actions as gesture inputs by using the information process of thefirst embodiment, and perform processes in accordance with theseactions.

Where a user performs an action of moving the hand up and down (fanningaction), as shown in FIG. 12, the portable device 1 calculates the handposition (the position with respect to the x-axis direction) by usingthe method of the first embodiment. That is, the portable device 1calculates the center-of-gravity position with respect to the directioncorresponding to the x-axis direction on the captured image (e.g., theup-down direction in FIG. 6). Thus, in other embodiments, the portabledevice 1 may no need to calculate the three-dimensional position of thehand, and may calculate the position with respect to at least onedirection. The portable device 1 may not need to calculate the shape ofthe hand.

The portable device 1 detects a fanning action based on the calculatedhand position, and performs an information process in accordance withthe fanning action. Specifically, the portable device 1 determineswhether or not a fanning action has been performed in accordance withthe change in the calculated hand position. That is, where thecalculated hand position repeatedly moves in the up-down direction, itis determined that a fanning action has been performed. If it isdetermined that a fanning action has been performed, the portable device1 performs a process of moving the object in the virtual space, forexample. The portable device 1 may calculate the changing speed of thecalculated position, and calculate the speed (rapidness) of the fanningaction in accordance with the changing speed. Then, the portable device1 may vary the content of the process in accordance with the speed ofthe fanning action. For example, the object moving speed may becontrolled in accordance with the speed of the fanning action.

Where a user performs an action of rotating the hand (rotating action),as shown in FIG. 13, the portable device 1 calculates thetwo-dimensional position of the hand (the position with respect to thex-axis direction and the y-axis direction) by using the method of thefirst embodiment. The portable device 1 detects a rotating action basedon the calculated two-dimensional position, and performs an informationprocess in accordance with the rotating action. Specifically, theportable device 1 determines whether or not a rotating action has beenperformed based on the path along which the calculated two-dimensionalposition moves on a two-dimensional coordinate system. For example, itis determined that a rotating action has been performed when the path isring-shaped. If it is determined that a rotating action has beenperformed, the portable device 1 performs a process of moving (e.g.,rotating) an object in the virtual space. For example, the process maybe a process of spinning a wheel of a monocycle that the playercharacter rides, or a process of winding a reel in a fishing game. Notethat the portable device 1 may calculate the rotation angle of therotating action (e.g., the angle representing the direction toward thehand position with respect to the center position of the rotation), andperform a process in accordance with the rotation angle (e.g., a processof rotating the object in accordance with the rotation angle). Theportable device 1 may calculate the rotation speed of thetwo-dimensional position, and calculate the speed of the rotating actionin accordance with the rotation speed. Then, the portable device 1 mayvary the content of the process in accordance with the speed of therotating action. For example, the rotation speed of the object may becontrolled in accordance with the speed of the rotating action.

Note that a fanning action with respect to the up-down direction (thex-axis direction) and a rotating action about an axis extending in theleft-right direction (the z-axis direction) have been described above asexamples. Now, according to the method for calculating the hand positionin the first embodiment, since it is possible to calculate thethree-dimensional position of the hand, it is possible to detect thefanning action and the rotating action with respect to other directions.Therefore, in other embodiments, the portable device 1 may detect afanning action with respect to another direction, or a rotating actionabout an axis extending in another direction.

In other embodiments, the portable device 1 may detect both the fanningaction and the rotating action. Specifically, the portable device 1 mayspecify the shape of the hand using the method of the first embodiment,and switch between actions to be detected in accordance with the shapeof the hand. For example, when the shape of the hand is specified asbeing open (the open-palm shape), the portable device 1 may detect thepresence/absence of a fanning action, and when the shape of the hand isspecified as being closed (the closed-fist shape), the portable device 1may detect the presence/absence of a rotating action.

Where a user performs the action of moving the hand in a predetermineddirection (the z-axis direction in FIG. 14) and the action of flicking afinger (flicking action) as shown in FIG. 14, the portable device 1determines the shape of the hand, while calculating the hand position(the position with respect to the z-axis direction) by using the methodof the first embodiment. In the example shown in FIG. 14, the portabledevice 1 uses the determination range of the first embodiment todetermine whether or not the hand is in a closed shape or in such ashape that the user has flicked a finger. When the shape of the handchanges from the closed shape to the flicking shape (the shape of thehand with one finger extended), the portable device 1 determines that aflicking action has been performed.

The portable device 1 may perform a process of moving an object in thevirtual space in accordance with the hand position, while making theobject perform a predetermined action in response to the flickingaction. For example, the portable device 1 may move a cannon object inthe virtual space in accordance with the hand position and fire acannonball from the cannon in response to a flicking action.

Second Embodiment

Next, a second embodiment will be described. In the second embodiment,the portable device 1 calculates the three-dimensional position of thehand based on the captured image by using a method different from thatof the first embodiment. The portable device 1 controls the movement ofthe cursor in the three-dimensional virtual space by using thecalculated three-dimensional position. Note that the hardwareconfiguration of the portable device 1 of the second embodiment will notbe described in detail as it is the same as that of the firstembodiment.

[1. Outline of Process Performed by Portable Device 1]

(Calculating Position and Shape of Hand)

In the second embodiment, the portable device 1 detects the area of thehand of the user included in a captured image. FIG. 15 shows an exampleof the area of the hand detected from a captured image. A hand area (thearea of the image of the hand in the captured image) 31 is detected by apredetermined image recognition process performed on the captured image.The image recognition process may be a conventional process including,for example, a contour extraction process, a process of detecting thehand area by pattern matching, a process of removing noise areas (areasother than the hand), etc. Note that it is possible to specify the shapeof the hand (e.g., whether the hand is in a closed shape or in an openshape) by an image recognition process of detecting the hand area 31 aswill be described later. The area of a finger in the hand area 31 can bespecified by detecting the hand area 31.

Upon detecting the hand area 31, the portable device 1 calculates thethree-dimensional position of the hand based on the hand area 31. Thespecific method for calculating the three-dimensional position of thehand may be any method. In the second embodiment, the portable device 1calculates the position with respect to the two-dimensional direction(the x-axis direction and the y-axis direction) perpendicular to theimage-capturing direction of the infrared camera 4, based on theposition of the hand area 31 on the captured image. The two-dimensionalposition of the hand (the position with respect to the x-axis directionand the y-axis direction) is calculated as the center position of thehand area 31 (the position P2 shown in FIG. 15). Note that in otherembodiments, the portable device 1 may calculate, as the two-dimensionalposition of the hand, the center-of-gravity position obtained with thebrightness value of each pixel in the hand area 31 being used as aweight. Note that the center-of-gravity position is a center-of-gravityposition obtained by a calculation performed for the hand area 31, andis different from the center-of-gravity position of the firstembodiment.

Based on the size of the hand area 31 on the captured image, theportable device 1 calculates the position (the distance from theinfrared camera 4) with respect to the image-capturing direction of theinfrared camera 4 (the z-axis direction). The method for calculating thehand distance (the position with respect to the z-axis direction) fromthe hand area 31 may be any method. For example, the portable device 1may store, in advance, information representing the correlation betweenthe size of the hand area 31 and the distance (which may be a table ormay be a function), and calculate the distance based on the correlation.

In the second embodiment, the portable device 1 specifies the shape ofthe hand based on the captured image. FIG. 15 shows the hand areaextracted from the captured image where the hand is in an open shape. Onthe other hand, FIG. 16 shows the hand area extracted from the capturedimage where the hand is in a closed shape. In the second embodiment, theportable device 1 specifies the shape of the hand to be either anopen-hand shape (the open-palm shape) or a closed-hand shape (theclosed-fist shape).

The method for specifying the shape of the hand may be any method. Forexample, the shape of the hand can be specified by an image recognitionprocess for detecting the hand area 31. In the second embodiment, theportable device 1 provides a reference image corresponding to theopen-hand shape and a reference image corresponding to the closed-handshape, and performs a pattern match using the two reference images inthe image recognition process so at to determine the shape of the hand.

As described above, in the second embodiment, the portable device 1calculates the three-dimensional position of the hand and the shape ofthe hand by detecting the area of the hand by an image recognitionprocess. Note that also in the second embodiment, the portable device 1may calculate the position and/or the shape of the hand by using themethod of the first embodiment.

(Cursor Control Based on Position and Shape of Hand)

In the second embodiment, based on the three-dimensional position of thehand calculated as described above, the portable device 1 controls thecursor in the three-dimensional virtual space. Based on the shape of thehand specified as described above, the portable device 1 switchesbetween the selected state and the unselected state of the cursor. Thus,in the second embodiment, the user can three-dimensionally move thecursor by three-dimensionally moving the hand, and can also perform aso-called “drag and drop” operation using the cursor.

FIG. 17 shows an example of an operation performed in the secondembodiment. As shown in FIG. 17, the user can move a cursor 33 inthree-dimensional directions in the virtual space displayed on thedisplay 2, by moving the hand, in three-dimensional directions, i.e., upand down (the x-axis direction), left and right (the z-axis direction),and forward and backward (the y-axis direction). That is, the portabledevice 1 can calculate the three-dimensional position of the cursor 33in the virtual space based on the three-dimensional position of the handcalculated based on the captured image. The specific method forcalculating the position of the cursor 33 may be any method. Theportable device 1 may calculate the position of the cursor 33 so thatthe three-dimensional position of the cursor 33 in the virtual spacecorresponds to the three-dimensional position of the hand in the realspace. For example, the portable device 1 may calculate the position ofthe cursor 33 so as to move the cursor 33 in a direction (in the virtualspace) corresponding to the three-dimensional position of the hand withrespect to the reference position in the real space.

Now, in the present embodiment, the movement of the cursor 33 iscontrolled so that the moving direction of the hand generally coincideswith the moving direction of the cursor 33 on the display 2 (see thearrows in FIG. 17). That is, the portable device 1 calculates thethree-dimensional position including (a) the first position with respectto the side surface direction (the z-axis direction), (b) the secondposition with respect to the direction (the y-axis direction)perpendicular to the side surface direction and perpendicular to thescreen of the display 2, and (c) the third position with respect to thedirection (the x-axis direction) perpendicular to the side surfacedirection and parallel to the screen. Then, the portable device 1 (a)performs a cursor movement process with respect to the direction in thevirtual space corresponding to the left-right direction of the screenbased on the first position, (b) performs a cursor movement process withrespect to the direction in the virtual space corresponding to the depthdirection of the screen based on the second position, and (c) performs acursor movement process with respect to the direction in the virtualspace corresponding to the up-down direction of the screen based on thethird position. Then, the moving direction of the hand of the user inthe real space generally coincides with the moving direction of thecursor 33 in the virtual space displayed on the display 2, therebyenabling an intuitive operation and improving the controllability of thecursor 33.

The portable device 1 switches between the selected state and theunselected state of the cursor 33 in accordance with the shape of thehand specified based on the captured image. As used herein, the selectedstate refers to a state where the cursor 33 can select an object in thevirtual space, and the unselected state refers to a state where thecursor 33 cannot select an object. When the cursor 33 is selecting anobject, the object is moved following the movement of the cursor 33. Inthe second embodiment, the cursor 33 is set to the selected state whenthe hand in the closed shape is detected, and the cursor 33 is set tothe unselected state when the hand in the open shape is detected.

FIG. 18 shows an example where an object in the virtual space is movedby using the cursor 33. FIG. 18 shows how after the cursor 33 is movedin the unselected state, the state switches to the selected state whilethe cursor 33 points at an object 34, and the cursor 33 in the selectedstate moves together with the object 34. In FIG. 18, the user firstmoves the hand in the open position. In response to this, the portabledevice 1 moves the cursor 33 in the unselected state. The user moves thehand in the open position until the cursor 33 is located so as to pointat the object 34. When the cursor 33 is located so as to point at theobject 34, the user closes the hand. In response to this, the portabledevice 1 switches the cursor 33 from the unselected state to theselected state. Then, the object 34 pointed at by the cursor 33 becomesselected. Then, the user moves the hand while keeping the hand in theclosed position. In response to this, the portable device 1 moves theobject 34 together with the cursor 33. Thus, the user can perform a dragoperation on the object 34. Then, in response to the user changing thehand position from the closed position to the open position, the cursor33 turns into the unselected state, thereby stopping the movement of theobject 34. Thus, the user can perform a drop operation on the object 34.

As described above, according to the second embodiment, the user canperform a drag and drop operation in the virtual space by way of anaction of three-dimensionally moving the hand and an action of switchingthe hand position between the closed position and the open position. Asthe user moves a hand in the space in the side surface direction of theportable device 1, the user can perform a cursor operation while theuser can easily view the display since the hand is unlikely to get inthe way of the user viewing the display.

Note that in other embodiments, the cursor may move only intwo-dimensional directions. That is, the portable device 1 may calculatethe hand position in predetermined two-dimensional directions, andcalculate the two-dimensional position of the cursor in the virtualspace based on the calculated two-dimensional position.

[2. Specific Example of Process Performed by Portable Device 1]

Next, a specific example of the process performed by the portable device1 of the second embodiment will be described. FIG. 19 is a flow chartshowing an example of the flow of the process performed by the portabledevice 1 of the second embodiment. A series of processes shown in FIG.19 may be performed under any condition and with any timing. Forexample, the series of processes are performed while running apredetermined application program (e.g., a game program) that acceptsgesture inputs.

In step S10, as with the process of step S1 of the first embodiment, thecontrol section 14 obtains, from the infrared camera 4, the capturedimage captured by the infrared camera 4. Then, in step S11, the controlsection 14 detects the hand area in the captured image obtained in stepS11. Moreover, in step S12, the control section 14 calculates thethree-dimensional position of the hand based on the hand area detectedin step S11. Then, in step S13, the control section 14 specifies theshape of the hand based on the captured image obtained in step S10. Theseries of processes of steps S11 to S13 are performed by the methoddescribed in “(Calculating position and shape of hand)” above.

In step S14, the control section 14 determines whether or not the cursor33 is in the selected state. This determination is made based on whetheror not the shape of the hand specified in step S13 is an open shape. Ifthe determination result of step S14 is affirmative, the process of stepS15 is performed. On the other hand, if the determination result of stepS14 is negative, the process of step S16 to be described later isperformed.

In step S15, the control section 14 determines whether or not an objectis being selected by the cursor 33. This determination is made based onwhether or not the cursor 33 is located so as to point at the object. Ifthe determination result of step S15 is negative, the process of stepS16 is performed. On the other hand, if the determination result of stepS15 is affirmative, the process of step S17 to be described later isperformed.

In step S16, the control section 14 controls the movement of the cursor33 in the virtual space based on the three-dimensional position of thehand calculated in step S12. The control of the movement of the cursor33 is performed by the method described in “(Cursor control based onposition and shape of hand)” above. Following step S16, the process ofstep S18 to be described later is performed.

On the other hand, in step S17, the control section 14 controls themovement of the cursor 33 in the virtual space and the object beingpointed at by the cursor 33, based on the three-dimensional position ofthe hand calculated in step S12. The method for controlling the movementof the cursor 33 and the object is the same as the movement controlmethod of step S16 described above, except that the object is movedtogether with the cursor 33. Following step S17, the process of step S18to be described later is performed.

In step S18, the control section 14 generates an image of the virtualspace representing the results of the process of step S16 or S17, anddisplays the generated image on the display 2. Then, in step S19, thecontrol section 14 determines whether or not to end the cursor movementcontrol based on the position and the shape of the hand. The specificmethod of the determination in step S19 may be any method. In the secondembodiment, if a user gives a predetermined end instruction (e.g., aninstruction to end an application), it is determined to end the cursormovement control process. If the determination result of step S19 isaffirmative, the control section 14 ends the process shown in FIG. 19.On the other hand, if the determination result of step S19 is negative,the control section 14 performs the process of step S10 again.Thereafter, the series of processes of steps S10 to S19 are performedrepeatedly until it is determined in step S19 to end the process.

[3. Function/Effect of Second Embodiment]

As described above, in the second embodiment, the portable device 1includes the infrared camera 4 for capturing an image (of the space) inthe side surface direction of the housing, and calculates thethree-dimensional position of an object (a hand of a user) included inthe captured image based on the captured image captured by the infraredcamera 4 (step S12). Then, the portable device 1 performs an informationprocess in accordance with the three-dimensional position (step S16,S17). As described above, the portable device 1 performs an informationprocess in which a user's action of moving a hand, etc., is used as aninput. Since the user can perform input operations by moving a hand,etc., it is possible to make a variety of inputs. Since the infraredcamera 4 captures an image of the space in the side surface direction ofthe portable device 1, and the user only needs to move a hand, etc., inthe space in the side surface direction of the portable device 1, theuser can hold the portable device 1 in one hand and perform an inputoperation using the other hand (see FIG. 5). Therefore, the user caneasily perform input operations.

Note that also in the first embodiment, as in the second embodiment, thethree-dimensional position of an object is calculated, and aninformation process in accordance with the calculated three-dimensionalposition is performed. Therefore, also in the second embodiment, as inthe first embodiment, such effects as those described above can berealized.

In the second embodiment (this also applies to the first embodiment),the portable device 1 specifies the shape of the object (a hand) basedon the captured image (step S13), and performs an information process inaccordance with the three-dimensional position and the shape of theobject (step S14). As described above, the user can perform inputoperations based on the shape of the hand, thereby allowing for a widervariety of inputs.

In the second embodiment (this also applies to the first embodiment),the portable device 1 specifies the position (the position of thecursor) in the three-dimensional virtual space based on thethree-dimensional position. Then, the user can specify athree-dimensional position in the virtual space through an operation ofthree-dimensionally moving the hand. Moreover, for the specifiedposition, the portable device 1 performs different information processesdepending on the shape of the object (a hand) (e.g., performs processeswhile switching between the selected state and the unselected state ofthe cursor (step S14, S16, S17)). As described above, the user canspecify a position in the virtual space by moving the hand, and specifythe content of the information process based on the shape of the hand,thereby providing a position-specifying operation with a goodcontrollability for the user.

Note that in the second embodiment, the portable device 1 specifies animage area of an object (a hand) included in the captured image (stepS11), and specifies the shape of the object based on the image area(step S13). Then, the portable device 1 calculates the position of theobject based on the specified image area of the object (step S12). Then,the portable device 1 can calculate the position and the shape of theobject by specifying the area of the object.

Note that the portable device 1 may calculate the two-dimensionalposition of the object on the captured image, and calculate thethree-dimensional position based on the two-dimensional position (e.g.,calculate the three-dimensional position including the two-dimensionalposition), as in the first embodiment. The portable device 1 may specifythe shape of the object based on the calculated two-dimensional position(e.g., specify the shape by using a determination range which is setbased on the two-dimensional position). Then, the portable device 1 canspecify the position and the shape of the object by a simple process.

In the second embodiment (this also applies to the first embodiment),the portable device 1 includes the display 2 provided on the frontsurface of the housing, and performs a information process of displayingan image in accordance with the three-dimensional position on thedisplay 2 (step S18). Then, the results of the process performed byuser's hand-moving actions are displayed on the display. Now, since theuser moves a hand in the space in the side surface direction of theportable device 1, the hand is unlikely to get in the way of the imagedisplayed on the display, thus improving the viewability of the image.

[4. Variation]

(Variation in which Direction of Hand is Calculated)

Where the hand area in the captured image is detected as in the secondembodiment, the portable device 1 may calculate the direction of thehand. Now, the direction of the hand with respect to the rotation aboutan axis extending in the image-capturing direction (the z-axisdirection) can be calculated based on the direction of the hand area inthe captured image. The direction of the hand with respect to therotation about an axis extending in the direction perpendicular to theimage-capturing direction (the x-axis direction or the y-axis direction,etc.) can be calculated based on the brightness value of the hand areain the captured image. An example will now be described, as a variationof the second embodiment, in which the direction of the hand iscalculated with respect to the rotation about an axis extending in thedirection perpendicular to the image-capturing direction.

FIG. 20 shows an example of how a user performs a direction changingoperation according to a variation of the second embodiment. As shown inFIG. 20, according to the present variation, the user performs anoperation (direction changing operation) of changing the direction ofthe hand in the rotational direction about an axis extending in theup-down direction (the x-axis direction), with the index fingerextended. The portable device 1 calculates the direction of the handbased on the captured image, and performs an information process inaccordance with the direction of the hand.

FIG. 21 shows an example of an area of a hand detected from a capturedimage for two cases where the hand is oriented differently. In FIG. 21,the brightness value for the hand area 31 detected from the capturedimage is represented by degrees of shading.

As described above, the brightness value of the object captured by theinfrared camera 4 is higher at positions closer to the infrared camera4. Therefore, if the index finger as seen from the user is orientedleftward with respect to the front surface direction (the negativey-axis direction), as shown in the state (a) of FIG. 20, i.e., if thefingertip of the hand is on the near side with respect to the wrist asseen from the infrared camera 4, the hand area 31 in the captured imagehas a brightness value distribution as shown in (a) of FIG. 21. That is,the brightness value is higher in the left-side area (fingertip) of thehand area 31 than in the right-side area (wrist). On the other hand, ifthe index finger as seen from the user is oriented rightward withrespect to the front surface direction, as shown in the state (b) ofFIG. 20, i.e., if the fingertip of the hand is on the far side withrespect to the wrist as seen from the infrared camera 4, the hand area31 in the captured image has a brightness value distribution as shown in(b) of FIG. 21. That is, the brightness value is lower in the left-sidearea (fingertip) of the hand area 31 than in the right-side area(wrist). Although not shown in the figure, if the index finger isoriented in the front surface direction, i.e., the fingertip and thewrist are at substantially the same distance as seen from the infraredcamera 4, the brightness value in the left-side area (fingertip) of thehand area 31 is substantially the same as the brightness value in theright-side area (wrist).

As described above, the direction of the hand with respect to therotation about the x axis can be calculated based on the difference inthe brightness value between the left-side area and the right-side areaof the captured image. According to the present variation, the portabledevice 1 specifies the left-side area position and the right-side areaposition from within the hand area 31 detected from the captured image,and compares the brightness value between the two positions specified.Note that the two positions to be specified may be any positions thatare selected from the hand area 31. For example, the portable device 1may specify a position at a predetermined distance from the left end ofthe hand area, and a position at a predetermined distance from the rightend of the hand area.

Based on the brightness value of the left-side area and the brightnessvalue of the right-side area, the portable device 1 calculates thedirection of the hand with respect to the rotation about an axisextending in the up-down direction (the x-axis direction). That is,where the brightness value of the left-side area and the brightnessvalue of the right-side area are generally the same (where thedifference is less than a predetermined value), the portable device 1determines that the hand is oriented in the front surface direction asseen from the user. If the brightness value of the left-side area ishigher than the brightness value of the right-side area by apredetermined value or more, the portable device 1 determines that thehand is oriented leftward with respect to the front surface direction(as seen from the user). If the brightness value of the left-side areais lower than the brightness value of the right-side area by apredetermined value or more, the portable device 1 determines that thehand is oriented rightward with respect to the front surface direction(as seen from the user). Thus, according to the present variation, theportable device 1 can determine whether or not the hand is oriented inthe front surface direction, leftward with respect to the front surfacedirection, or rightward with respect to the front surface direction. Inother embodiments, the portable device 1 may calculate the anglerepresenting the direction of the hand.

Note that while a case where the direction of the hand with respect tothe rotation about an axis extending in the up-down direction (thex-axis direction) is calculated has been described above, it is alsopossible to calculate the direction of the hand with respect to therotation about an axis extending in the front-rear direction (the y-axisdirection) by a method similar to that described above. That is, theportable device 1 may calculate the direction of the hand with respectto the rotation about an axis extending in the front-rear directionbased on the brightness value of the upper area of the hand area 31 andthe brightness value of the lower area thereof.

The portable device 1 performs an information process in accordance withthe direction of the hand calculated as described above. The informationprocess may be of any content. For example, the portable device 1performs an information process for controlling the orientation(attitude) of the object in the virtual space in accordance with thedirection of the hand. The portable device 1 may calculate the handposition in addition to the direction of the hand. Then, the portabledevice 1 may control the orientation (attitude) of the object in thevirtual space in accordance with the direction of the hand, and controlthe position of the object in accordance with the hand position.

Third Embodiment

Next, a third embodiment will be described. Where operations areperformed by the user by moving a hand, etc., around the portable device1, it may be difficult for the user to grasp the hand position formaking an intended input, or the positional relationship between theinfrared camera 4 (the portable device 1) and the hand. In view of this,in the third embodiment, the portable device 1 displays a guide imagefor representing the hand position. Note that the hardware configurationof the portable device 1 of the third embodiment will not be describedin detail as it is the same as that of the first embodiment.

[1. Outline of Process Performed by Portable Device 1]

FIG. 22 shows an example of an operation performed in the thirdembodiment. As shown in FIG. 22, in the third embodiment, the usercontrols an object (player character) 41 in the virtual space by movinga hand in the image-capturing direction (the z-axis direction). That is,the portable device 1 calculates the hand position with respect to theimage-capturing direction and controls the movement of the object 41.

Note that while FIG. 22 is directed to an example where a user controlsa player character in a racing game, the information process to beperformed with hand actions being used as inputs may be any informationprocess. In the third embodiment, the portable device 1 controls thedegree of turn (the left-right movement) of the object 41 in accordancewith the hand position with respect to the image-capturing direction(distance) (see the arrows shown in FIG. 22). The portable device 1 mayfurther control the speed of the object, and the like, through handactions (e.g., the hand position with respect to the y-axis direction,and the shape of the hand).

As shown in FIG. 22, a guide image 42 is displayed, in addition to animage representing the results of a process based on a user's operation(an image of the game space), on the display 2. The details of the guideimage 42 will be described later.

(Outline of Process Based on Hand Position)

FIG. 23 shows an example of the relationship between the hand positionand the information process. Since the portable device 1 calculates thehand position based on the captured image, it is not possible to detectthe hand and it is not possible to calculate the hand position if thehand is too far away from the infrared camera 4 (e.g., if the distanceto the infrared camera 4 is more than 40 cm). Herein, the range in whichthe hand position can be calculated is referred to as the “calculatablerange”, and the range in which the hand position cannot be calculated isreferred to as the “un-calculatable range”. As shown in FIG. 23, in thethird embodiment, the calculatable range is a range within apredetermined distance from the infrared camera 4, and theun-calculatable range is a range further away from the infrared camera 4than the predetermined distance.

In the third embodiment, the portable device 1 uses a predeterminedrange within the calculatable range, as the process range. The processrange is a range in which the content of the process changes inaccordance with the calculated hand position. That is, the portabledevice 1 changes the content of the process (herein, the degree of turnof the object) in accordance with the change in the hand position withinthe process range. Specifically, in the third embodiment, the portabledevice 1 makes the object 41 move straight when the calculated handposition is at a predetermined reference position within the processrange, and controls the degree of turn of the object 41 in accordancewith the difference between the calculated hand position and thereference position. As shown in FIG. 23, the object 41 is controlled toturn left when the calculated hand position is on the left side (closerto the infrared camera 4) of the reference position as seen from theuser, and the object 41 is controlled to turn right when the calculatedhand position is on the right side (away from the infrared camera 4) ofthe reference position as seen from the user. Note that the referenceposition is at the center of the process range in the third embodiment,but the reference position may be any position within the process range.

When the calculated hand position is outside the process range, theportable device 1 sets the degree of turn of the object 41 to a degreeof turn that would be used when the hand position is at one of theboundaries of the process range that is closer to the calculated handposition. That is, when the hand position moves from within the processrange to the outside of the process range, the degree of turn of theobject 41 does not increase.

Note that as shown in FIG. 23, a prior warning range is set within thecalculatable range in the third embodiment. The prior warning range isset for the purpose of giving the user a prior warning before the handenters the un-calculatable range. The prior warning range is set as arange that is away from a predetermined position within the calculatablerange (e.g., the reference position described above) by a predetermineddistance toward the un-calculatable range. Note that it can also be saidthat the prior warning range is a range further away from the positionof the infrared camera 4 than a predetermined distance. While the priorwarning range does not overlap the process range in FIG. 23, it mayoverlap the process range in other embodiments.

In other embodiments, the portable device 1 may not only calculate thehand position with respect to the image-capturing direction (distance)but also calculate the hand position with respect to another direction.

(Specific Example of Guide Image)

FIG. 24 shows an example of a guide image. As shown in FIG. 24, theguide image 42 includes a range image 45 and an indicator image 46. Therange image 45 represents the area of the process range in the realspace. The shape of the range image 45 may be any shape. In the thirdembodiment, the range image has a bar shape extending in the horizontaldirection so as to represent the process range for the position in theimage-capturing direction (the left-right direction as seen from theuser).

The indicator image 46 is an image representing the hand position. Theindicator image 46 moves, in accordance with the calculated handposition, within a range that is represented by the range image 45. Notethat in the third embodiment, the output (the degree of turn of theobject 41) of the portable device 1 changes in accordance with thecalculated hand position, as described above. Therefore, it can be saidthat the position of the indicator image 46 changes in accordance withthe output.

Now, with a control method in which the hand is moved around theportable device 1, as in the third embodiment, the hand canadvantageously be moved freely, but it may be difficult for the user tograsp the reference position. In contrast, in the third embodiment, itis easier for the user to grasp the hand position because of the guideimage 42. That is, the user can grasp the relationship between the handposition and the output by seeing the guide image 42, making it easierfor the user to perform operations.

Note that in the third embodiment, the indicator image 46 is an imagerepresenting the hand (of the user), which is assumed to be an objectused for performing operations. Thus, the user can easily know that theindicator image 46 is representing an object used for performingoperations. The user can also easily know the method of operation (i.e.,performing operations by using a hand).

Note that in other embodiments, the portable device 1 may change thedisplay mode of the indicator image 46 in accordance with the specifiedshape of the hand. For example, the portable device 1 may display anindicator image that represents the specified shape of the hand.Specifically, where the portable device 1 determines that whether thehand is in a closed shape or in an open shape, the portable device 1 maydisplay an indicator image representing the closed-fist shape when theclosed-hand shape is specified and display an indicator imagerepresenting the open-palm shape when the open-hand shape is specified.

As shown in FIG. 24, the guide image 42 includes a process referenceimage 47. The process reference image 47 represents the position atwhich the indicator image 46 is placed when the hand is located at thereference position. That is, when the calculated hand position is thereference position, the indicator image 46 is placed at the positionrepresented by the process reference image 47. Therefore, the user canmake the object 41 move straight by placing the hand so that theindicator image 46 is located at the process reference image 47. Notethat when the reference position is changed (e.g., the referenceposition may be changed in response to a user's settings-changingoperation), the position of the process reference image 47 is alsochanged.

As described above, the portable device 1 controls the output so as togive an output in accordance with the relationship between thecalculated hand position and the predetermined reference position, anddisplays the guide image including the process reference image on thedisplay 2. Then, the user can adjust the hand so that the hand is placedat the reference position by seeing the guide image, thereby making iteven easier for the user to perform operations.

As shown in FIG. 24, the guide image 42 includes a camera referenceimage 48. The camera reference image 48 is displayed so as to representthe positional relationship between the area of the process rangerepresented by the range image 45 and the infrared camera 4 (theportable device 1). That is, the camera reference image 48 is placed atsuch a position that the positional relationship between the range image45 and the camera reference image 48 corresponds to the positionalrelationship between the area of the process range and the infraredcamera 4 (the portable device 1). Note that it can also be said that thecamera reference image 48 is displayed so as to represent the positionalrelationship between the hand of the user and the infrared camera 4 (theportable device 1). That is, it can be said that the camera referenceimage 48 is placed at such a position that the positional relationshipbetween the indicator image 46 and the camera reference image 48corresponds to the positional relationship between the hand of the userand the infrared camera 4 (the portable device 1).

In the third embodiment, since the infrared camera 4 is present on theleft side (the negative z-axis direction) of the process range, thecamera reference image 48 is displayed on the left side of the rangeimage 45 as shown in FIG. 24. With the camera reference image 48, theuser can easily know about where the hand should be placed with respectto the infrared camera 4 (the portable device 1).

Note that while the guide image 42 is displayed at an upper leftposition on the screen of the display 2 in FIG. 22, the position atwhich the guide image 42 is displayed may be any position. For example,the guide image 42 may be displayed in an area on the side where theinfrared camera 4 is provided (in other words, on the side of theimage-capturing direction of the infrared camera 4). That is, in thepresent embodiment, the guide image 42 may be displayed in an area onthe right side of the center of the screen. Then, the user can easilyview the guide image 42 while moving the hand, and it is easier for theuser to check the relationship between the display of the guide image 42(the position of the indicator image 46 with respect to the range image45) and the hand.

FIG. 25 shows an example of a guide image to be displayed when the handposition is in the prior warning range. In the third embodiment, whenthe hand position is in the prior warning range, the portable device 1changes the display mode of the indicator image 46 (as compared with acase where the hand position is not in the prior warning range but is inthe calculatable range). While the color of the indicator image 46 ischanged in FIG. 25, the size and/or the shape of the indicator image 46may be changed in other embodiments.

Thus, as the position of the hand of the user moves from the processrange and comes close to the un-calculatable range, the hand position isincluded in the prior warning range, thereby changing the display modeof the indicator image 46. Therefore, as described above, when the userattempts to move the hand position into the un-calculatable range, theportable device 1 can notify the user in advance of the hand positionentering the un-calculatable range, by changing the indicator image 46.Then, it is possible to reduce the possibility of the user moving thehand position into the un-calculatable range, thus improving thecontrollability for operations using a hand.

Note that in the third embodiment, since the prior warning range is seton the right side of the process range, when the hand position is in theprior warning range, it means that the hand position has moved past theright end of the process range. Therefore, in the third embodiment, whenthe hand position is in the prior warning range, the indicator image 46is displayed at the right end of the range image 45. Thus, when the handposition is in the prior warning range, the indicator image 46 isdisplayed at the end portion of the range image 45 on the side of theprior warning range with respect to the process range. Thus, it ispossible to notify the user, in an easy-to-understand manner, of whichside the hand position is off the process range.

Note that in other embodiments, the prior warning range may be set onboth sides of the process range. For example, where the hand position isregarded un-calculatable when it is too close to the infrared camera 4,the prior warning range may also be set on one side of the process rangethat is closer to the infrared camera 4.

FIG. 26 shows an example of a guide image to be displayed when the handposition is un-calculatable. In the third embodiment, when the handposition is in the un-calculatable range (when the hand position isun-calculatable), the portable device 1 changes the display mode of theindicator image 46, as compared with that when the hand position iscalculated. Specifically, as shown in FIG. 26, when the hand position isin the un-calculatable range, the portable device 1 changes theindicator image 46 into a shape that indicates that the hand cannot bedetected (a question-mark shape in FIG. 26). Thus, it is possible tonotify the user, in an easy-to-understand manner, of the hand positionbeing in such a state that the hand position cannot be detected(calculated). Note that while the shape of the indicator image 46 ischanged in FIG. 26, the size and/or the color of the indicator image 46may be changed in other embodiments. The display mode of the indicatorimage 46 when the hand position is in the un-calculatable range isdifferent from the display mode when the hand position is in the priorwarning range.

When the hand position is in the un-calculatable range, the indicatorimage 46 is displayed at the end portion of the range image 45 on theside of the un-calculatable range with respect to the process range.Therefore, in the third embodiment, the indicator image 46 is placed atthe end portion of the range image 45 in such a case, as in the priorwarning range. Note however that in other embodiments, when the handposition is in the un-calculatable range, the indicator image 46 may beplaced at a different position (e.g., the position indicated by theprocess reference image 47).

Note that in the present embodiment, the portable device 1 changes thedisplay mode of the indicator image 46 when the hand position is in theprior warning range or the un-calculatable range. Now, in otherembodiments, the portable device 1 in such a case may change the displaymode of a part or whole of the guide image 42, as opposed to onlychanging the display mode of the indicator image 46. For example, theportable device 1 in such a case may change the display mode of therange image 45 and/or the process reference image 47.

(Timing for Displaying Guide Image)

In the present embodiment, the portable device 1 displays the guideimage 42, together with the game image, during the game process (seeFIG. 22). Therefore, by seeing the guide image 42 while playing thegame, the user can check the relationship between the hand position andthe output, making it easier for the user to perform operations.

Note that in other embodiments, the timing for displaying the guideimage 42 may be any timing. For example, the guide image 42 may bedisplayed before the game is played. That is, the portable device 1displays the guide image 42 before the start of a game that is played inaccordance with operations performed with a hand, and moves theindicator image in accordance with the movement of the hand, as in theembodiments described above. Then, in response to satisfaction of apredetermined starting condition (e.g., the user giving an instructionto start the game, or the passage of a predetermined amount of timesince when the guide image 42 is displayed, etc.), the portable device 1starts the game process that is played in accordance with operationsperformed with a hand. By displaying the guide image 42 before the startof the game, as described above, the user can grasp, to some degree, thefeel of the game operation using a hand before the start of the game,making it easier for the user to perform game operations. Note thatwhere the guide image 42 is displayed before the game is played, theguide image 42 does not need to be displayed during the game, or theguide image 42 may be displayed during a part or whole of the game.

[2. Specific Example of Process Performed by Portable Device 1]

Next, a specific example of the process performed by the portable device1 of the third embodiment will be described. FIG. 27 is a flow chartshowing an example of the flow of the process performed by the portabledevice 1 of the third embodiment. A series of processes shown in FIG. 27may be performed under any condition and with any timing. For example,the series of processes are performed while running a predeterminedapplication program (e.g., a game program) that accepts gesture inputs.

First, in step S21, the control section 14 obtains, from the infraredcamera 4, a captured image captured by the infrared camera 4, as in theprocess of step S1 of the first embodiment. Then, in step S22, thecontrol section 14 calculates the hand distance (the position withrespect to the z-axis direction) based on the captured image obtained instep S21. The method for calculating the hand distance may be anymethod, and the hand distance may be calculated by the calculationmethod of the first embodiment described above, for example. Note thatthe calculation of the hand distance may fail when the hand position isin the un-calculatable range or when the hand position is not includedin the captured image, as described above.

In step S23, the control section 14 performs an information processbased on the hand distance calculated in step S22. The informationprocess may be of any content. In the third embodiment, the controlsection 14 performs a game process in which the hand distance is used asan input. Specifically, the control section 14 performs a process ofcontrolling the degree of turn of the object 41 in accordance with thehand distance within the virtual game space. Note that if the handdistance is not calculated in step S22, the information processdescribed above is performed while it is considered that no input isbeing made.

In step S24, the control section 14 determines whether or not thecalculation of the hand distance has been successful in the process ofstep S22. If the determination result of step S24 is affirmative, theprocess of step S25 is performed. On the other hand, if thedetermination result of step S24 is negative, the process of step S28 tobe described later is performed.

In step S25, the control section 14 determines whether or not the handposition (distance) calculated in step S22 is included in the priorwarning range. If the determination result of step S25 is negative, theprocess of step S26 is performed. On the other hand, if thedetermination result of step S24 is affirmative, the process of step S27to be described later is performed.

In step S26, the control section 14 displays an image representing theprocess results of the information process in step S23 on the display 2,and displays the guide image 42 on the display 2 in a normal displaymode (see FIG. 22). Note that the normal display mode refers to thedisplay mode used when the hand position is in the calculatable rangeand not in the prior warning range (see FIG. 24). In the process of stepS26, the indicator image 46 is placed, on the range image 45, at aposition in accordance with the hand distance calculated in step S22.Following the process of step S26, the process of step S29 is performed.

In step S27, the control section 14 displays an image representing theprocess results of the information process in step S23 on the display 2,and displays the guide image 42 on the display 2 in a prior warningmode. Note that the prior warning mode refers to the display mode usedwhen the hand position is in the prior warning range (see FIG. 25). Inthe process of step S27, the indicator image 46 is placed at the rightend position on the range image 45. Following the process of step S27,the process of step S29 is performed.

In step S28, the control section 14 displays an image representing theprocess results of the information process of step S23 on the display 2,and displays the guide image 42 on the display 2 in such a display modeas to indicate the uncontrollability. Note that “such a display mode asto indicate the uncontrollability” refers to the display mode in a casewhere the hand position is in the un-calculatable range (see FIG. 26).In the process of step S28, the indicator image 46 is placed at theright end position on the range image 45. Following the process of stepS28, the process of step S29 is performed.

In step S29, the control section 14 determines whether or not to end theseries of information processes in which the hand distance is used as aninput. The specific method of the determination in step S29 may be anymethod. In the third embodiment, it is determined that the series ofinformation processes are to be ended when the user performs apredetermined end instruction (e.g., an instruction to end theapplication). If the determination result of step S29 is affirmative,the control section 14 ends the process shown in FIG. 27. On the otherhand, if the determination result of step S29 is negative, the controlsection 14 performs the process of step S21 again. Thereafter, theseries of processes of steps S21 to S29 are performed repeatedly untilit is determined in step S29 to end the process.

[3. Function/Effect of Third Embodiment]

As described above, in the third embodiment, hand-held informationprocessing device (the portable device 1) calculates the position of theobject (the hand of the user) included in the captured image based onthe captured image captured by the infrared camera 4 (step S22). Theinformation processing device gives an output in accordance with theposition of the object to an output device (the display 2) (step S23).The information processing device displays, on a predetermined displaydevice (the display 2), the guide image 42 including the range image 45representing a range and the indicator image 46 whose position changesin accordance with the change in the output within the range (step S26,S27).

The “output in accordance with the position of the object” may be animage display output such as that of the third embodiment, or a soundoutput from a speaker. That is, the portable device 1 may display theguide image on the display 2 in the case where the sound output ischanged in accordance with the hand position. The “predetermined displaydevice” may be a display device (the display 2) of the informationprocessing device, or a display device (e.g., a television) that isseparate from the information processing device.

As described above, the portable device 1 performs an informationprocess in which a user's operation of moving a hand, etc., is used asan input. That is, since the user can perform input operations by movinga hand, etc., it is possible to make a variety of inputs. As describedabove, since the portable device 1 displays the guide image, the usercan know the position of a hand, etc. Thus, it is possible to improvethe controllability for operations using a hand, etc.

Note that in the third embodiment, the infrared camera 4 captures animage of the space in the side surface direction of the housing of theportable device 1. Therefore, the user only needs to move a hand, etc.,in the space in the side surface direction of the portable device 1, andthe user can easily perform input operations.

The portable device 1 displays the guide image 42 including apredetermined device reference image (the camera reference image 48) onthe display 2 so that the positional relationship between the indicatorimage 46 and the predetermined device reference image corresponds to thepositional relationship between the hand and the infrared camera 4 (FIG.24). Then, the portable device 1 can present to the user about where thehand is with respect to the infrared camera 4 (the portable device 1).Thus, it is possible to improve the controllability for operations usinga hand, etc.

Note that it can also be said that the portable device 1 displays theguide image including a predetermined device reference image (the camerareference image 48) on the display 2 so that the positional relationshipbetween the range image 45 and the predetermined device reference imagecorresponds to the positional relationship between the area and theinfrared camera 4. Then, the portable device 1 calculates the positionof the object at least within a predetermined area (within the processrange shown in FIG. 23). Then, the portable device 1 can present to theuser about where the area in which operations using a hand, etc., can bemade is with respect to the infrared camera 4 (the portable device 1).Thus, it is possible to improve the controllability for operations usinga hand, etc.

When the position of the object cannot be calculated, the portabledevice 1 changes the display mode of the guide image, as compared withthat when the position of the object is calculated (step S28 shown inFIG. 26 and FIG. 27). Then, it is possible to notify the user that thehand position cannot be detected (calculated), and it is possible toimprove the controllability for operations using a hand, etc.

When the calculated position of the object is further away toward theun-calculatable area from the predetermined reference position by apredetermined distance or more (when the calculated position is withinthe prior warning range), the portable device 1 changes the display modeof the guide image, as compared with that when the calculated positionis within the predetermined distance (step S27 shown in FIG. 25 and FIG.27). As described above, when the user attempts to move the position ofthe object (a hand, etc.) into the un-calculatable range, the portabledevice 1 can notify the user in advance of the position of the objectentering the un-calculatable area. Then, it is possible to reduce thepossibility of the user moving the position of the object into theun-calculatable area, thus improving the controllability for operationsusing a hand, etc.

The portable device 1 calculates, based on the captured image, theposition of the object with respect to the image-capturing direction ofthe infrared camera 4. Then, possible methods for calculating theposition of the object include the method based on the brightness in thecaptured image (the first embodiment), and the method based on the sizeof the object in the captured image (the second embodiment). Now, if thecolor or the size of the object is not known (e.g., if the object is anobject that inherently has individual differences such as the hand ofthe user), it is difficult to accurately calculate the absolute position(distance) thereof from the infrared camera 4. Therefore, with thesemethods, it may be difficult for the user to know about where the usershould place the hand in order to obtain an intended output. Regardingthis problem, according to the above description, it is possible topresent to the user about where the user should place the hand in aneasy-to-understand manner, thus improving the controllability foroperations using a hand.

The portable device 1 controls the output so as to obtain an output thatis in accordance with the relationship between the calculated positionof the object and the predetermined reference position (FIG. 23). Then,the portable device 1 displays, on the display 2, the guide image 42including the process reference image 47 placed at a positioncorresponding to the reference position within the range represented bythe range image 45 (FIG. 24). Then, the user can adjust the hand so thatthe hand is placed at the reference position by seeing the guide image42, thereby making it even easier for the user to perform operations.

[4. Variation]

In the third embodiment, the portable device 1 detects a hand-movingoperation with respect to a predetermined direction so as to display aguide image representing a position with respect to the direction.Herein, the operation to be detected by the portable device 1 may be anyoperation, and the display mode of the guide image may be any displaymode. Variations of the guide image will now be described.

FIG. 28 shows an example of a variation of the guide image. For example,the portable device 1 may detect a rotating action described above inthe first embodiment, and may display the guide image shown in FIG. 28on the display 2. The guide image shown in FIG. 28 includes a rangeimage 51 and an indicator image 52. The range image 51 is a circulararea, and the indicator image 52 is a linear image placed on the rangeimage 51.

Note that as in the third embodiment, the guide image may be displayedas an image different from the image (the game image, etc.) representingthe output results of a rotating action. The guide image may bedisplayed as a part of an image representing the output results of arotating action. For example, where a game process is performed in whicha monocycle that the player character rides advances in accordance withthe rotating action, the guide image may be displayed as being the wheelof the monocycle.

In the present variation, the portable device 1 controls the display sothat the range image 51 and the indicator image 52 rotate in accordancewith the rotating action. For example, the portable device 1 calculatesthe rotation angle in accordance with the rotating action, and displaysthe range image 51 and the indicator image 52 rotating, wherein theindicator image 52 represents the calculated rotation angle. Thus, thedirection of the indicator image 52 represents the calculated rotationangle. Therefore, the user can know the rotation angle from thedirection of the indicator image 52. Note that in other embodiments, thedisplay may be controlled so that the range image 51 is fixed (i.e.,does not rotate in accordance with the rotating action) while only theindicator image 52 rotates in accordance with the rotating action.

As described above, the portable device 1 may calculate the direction ofthe object (rotation angle) based on the captured image, and produce anoutput in accordance with the direction (display the rotation of therange image 51). Then, the guide image may include an indicator imagewhose direction changes within the range of the range image (inaccordance with the change of the output). Then, the user can know thedirection of a hand, etc., from the guide image, thus improving thecontrollability for operations using a hand.

In other embodiments, the portable device 1 may calculate thetwo-dimensional position of the object (e.g., a hand), in which case theguide image may represent a two-dimensional range. That is, the guideimage may include a range image that represents a two-dimensional range,and an indicator image that represents the calculated two-dimensionalposition in the two-dimensional range.

In other embodiments, the range image included in the guide image mayalso serve as the object placed in the virtual space (e.g., the gamespace). FIG. 29 shows another example of a variation of the guide image.FIG. 29 shows a game image displayed on the display 2 by the portabledevice 1. In the example shown in FIG. 29, the portable device 1calculates the hand position of the user based on the captured image,and controls the movement of a cannon 55 along a rail 56 in the virtualspace in accordance with the calculated hand position.

Specifically, the portable device 1 calculates a target position on therail 56 based on the hand position, and moves the cannon 55 to thetarget position. Note that there is an upper limit to the moving speedof the cannon 55, and if the movement of the target position inaccordance with the hand position is faster than the maximum speed, thecannon 55 is controlled to move, lagging behind the movement of thetarget position.

Now, the portable device 1 displays, on the display 2, an indicatorimage 57 representing the target position on the rail 56 (see FIG. 29).Then, it can be said that the rail 56 is the range image. Thus, a part(the range image, etc.) of the guide image may be an image of an objectappearing in the virtual space. Note that also in the example shown inFIG. 29, as in the third embodiment, the indicator image 57 is an imagedifferent from the object (the cannon 55) controlled by the user.

In the embodiment described above, the portable device 1 displays theguide image when the hand position is calculated based on the capturedimage captured by an image-capturing unit (the infrared camera 4). Now,the method for detecting the object present at a position away from theportable device 1 and calculating the position (or the direction) of theobject based on the detection result may be any method. For example, inother embodiments, the portable device 1 may display the guide imagewhen the position (distance) of the hand is calculated based on thedetection result of the distance measuring sensor 5. That is, theportable device 1 may calculate the position of the object (a hand)based on the output of the distance measuring sensor 5, and control apredetermined output device so as to produce an output in accordancewith the position of the object. Then, the portable device 1 may displaya guide image including an indicator image of which the position changesin accordance with the change of the output. Also when the distancemeasuring sensor 5 is used, it is possible to improve thecontrollability for operations using a hand, etc., by means of the guideimage, as in the embodiments described above.

Fourth Embodiment

Next, a fourth embodiment will be described. In the fourth embodiment,the portable device 1 detects, based on a captured image, an operation(touch operation) of touching a side surface of the portable device 1.The portable device 1 performs various information processes (e.g., theprocess of starting a game, etc.) in response to the touch operation. Byallowing for such touch operations, it is possible to improve thecontrollability. Note that the hardware configuration of the portabledevice 1 of the fourth embodiment will not be described in detail as itis the same as that of the first embodiment.

FIG. 30 shows an example of how a user performs a touch operation on theportable device 1. As shown in FIG. 30, a touch operation is anoperation in which a hand of the user contacts the infrared camera 4. Itcan also be said that a touch operation is an operation in which (thelens portion of) the infrared camera 4 is covered with the hand. Notethat in the fourth embodiment, the portable device 1 detects a touchoperation using the brightness of the captured image (e.g., the averagevalue of the brightness of the entire captured image), the details ofwhich will be described later. Therefore, even if the hand is not incontact with the infrared camera 4, a touch operation may be detected ifthe hand is located in the vicinity of the infrared camera 4. That is,in the fourth embodiment, the touch operation does not require that thehand be in contact with the infrared camera 4 in a strict sense, but itonly requires that the hand be placed in the vicinity of the infraredcamera 4. For example, the infrared camera 4 may be placed at adepressed position with respect to the surrounding portion of thehousing 21 so that the hand of the user does not contact the infraredcamera 4. Then, the touch operation is an operation of contacting aportion of the housing 21 around the infrared camera 4 in a strictsense. In the present embodiment, such an operation will be regarded asan operation of contacting the infrared camera 4 with the hand, and willbe referred to as a touch operation.

[1. Outline of Process Performed by Portable Device 1]

(Method for Detecting Touch Operation)

Next, a method for detecting a touch operation will be described. FIG.31 shows an example of the transition through captured images that areobtained during a touch operation. Captured images that are obtainedfrom before a touch operation until when the touch operation isperformed will transition as shown in FIG. 31, for example.

The captured image shown in (a) of FIG. 31 is an image that is obtainedat a point in time before a touch operation is performed. At this point,the hand of the user is somewhat away from the infrared camera 4, andthe entire hand of the user is included in the captured image. In thiscase, the brightness of the captured image is a normal brightness (arelatively low brightness among FIGS. 31(a) to 31(c)).

The captured image shown in (b) of FIG. 31 is an image that is obtainedat a point in time immediately before a touch operation is performed. Atthis point, the hand of the user has come quite close to the infraredcamera 4, and the hand of the user is partly outside the captured image.In this captured image, the area of the hand of the user is larger thanthat of the captured image shown in (a) of FIG. 31, and the brightnessof the captured image is higher. That is, the brightness of the capturedimage is higher than the normal brightness (a medium brightness amongFIGS. 31(a) to 31(c)).

The captured image shown in (c) of FIG. 31 is an image that is obtainedat a point in time when a touch operation is performed. At this point,the hand of the user is in contact with the infrared camera 4, andtherefore the infrared camera 4 is substantially covered by the hand.Therefore, since (substantially) the entire area of the captured imageis the hand area, the brightness of the captured image is even higher(than that of (b) of FIG. 31). That is, the brightness of the capturedimage is even higher than the normal brightness (a relatively highbrightness among FIGS. 31(a) to 31(c)).

As described above, based on the brightness of the captured image, theportable device 1 can determine whether or not a touch operation hasbeen performed. Specifically, the portable device 1 calculates, as thebrightness of the captured image, the average brightness value amongpixels of the captured image. Then, when the brightness of the capturedimage is greater than or equal to a predetermined value, the portabledevice 1 determines that a touch operation has been performed.

Note that in other embodiments, the portable device 1 may calculate thehand distance by using the distance measuring sensor 5, for example.Then, the portable device 1 may determine whether or not the hand hascontacted the infrared camera 4 based on the detection result of thedistance measuring sensor 5, and may detect a touch operation inaccordance with the determination result.

(Information Process Based on Touch Operation)

Next, an example of an information process in which a touch operation isused as an input will be described. In the fourth embodiment, where theportable device 1 performs the game process, the touch operation is usedas an operation for starting the game.

FIG. 32 shows an example of the flow of a game to be played by using atouch operation. Note that while the fourth embodiment is directed to anexample where a racing game is played, any game may be played.

In the fourth embodiment, the game is started in response to a touchoperation ((a) of FIG. 32). That is, the portable device 1 starts thegame (race) in response to detecting a touch operation before the startof the game. Note that the portable device 1 may start the game upondetecting the touch operation, or may start a countdown in response todetecting a touch operation, with the game starting when the count iszero.

During the game (during a gameplay), actions performed with the hand ofthe user are used as inputs ((b) of FIG. 32). That is, the portabledevice 1 calculates the position (or the direction) of the hand of theuser based on the captured image, and performs an information process(game process) in accordance with the movement of the hand. In thefourth embodiment, the portable device 1 makes the object (playercharacter) move or act based on the calculated hand position.

When a predetermined game condition is satisfied in the game, the gameis over ((c) of FIG. 32). That is, when the game condition is satisfiedduring the game, the portable device 1 stops the game and displays, onthe display 2, a game image that indicates that the game is over.

If the user performs a touch operation while a game image indicating thegame is over is displayed, the game is started (resumed) ((d) of FIG.32). That is, the portable device 1 resumes the game upon detecting atouch operation while the game is over. Note that the resumption of thegame means to include starting the game in an arbitrary state. Forexample, the portable device 1 may resume the game from where the gameended, or may start the game from the beginning.

As described above, according to the fourth embodiment, the user canstart the game by a touch operation performed on the side surface of theportable device 1. Since the user takes the hand off the infrared camera4 after performing a touch operation, the hand of the user willnaturally be placed within the field of view of the infrared camera 4.That is, after performing a touch operation, the user naturally bringsthe hand to a position where game operations are performed using thehand, and the user can therefore smoothly perform game operationsfollowing the touch operation. Therefore, according to the fourthembodiment, it is possible to improve the controllability for operationswhen starting the game.

Note that in other embodiments, the information process to be performedin response to a touch operation may be of any content. For example, theportable device 1 may perform an operation of pausing or ending the gamein response to a touch operation. For example, the portable device 1 maymake an object (player character) perform a predetermined action inresponse to a touch operation. For example, the portable device 1 maymove the object in the virtual space in accordance with the handposition calculated based on the captured image, and make the objectperform a predetermined action in response to a touch operation.

[2. Specific Example of Process Performed by Portable Device 1]

Next, a specific example of the process performed by the portable device1 of the fourth embodiment will be described. FIG. 33 is a flow chartshowing an example of the flow of the process performed by the portabledevice 1 of the fourth embodiment. A series of processes shown in FIG.33 may be performed under any condition and with any timing. Forexample, the series of processes described above are performed whilerunning a predetermined application program (e.g., a game program) thataccepts touch operations.

First, in step S31, the control section 14 obtains, from the infraredcamera 4, a captured image captured by the infrared camera 4, as in theprocess of step S1 of the first embodiment. Then, in step S32, thecontrol section 14 calculates the hand position based on the capturedimage obtained in step S1. Herein, the method for calculating the handposition may be any method, and the hand position is calculated by thecalculation method of the first embodiment described above, for example.The calculated hand position may be a position with respect to apredetermined direction, or may be a two-dimensional orthree-dimensional position.

In step S33, the control section 14 calculates the brightness of thecaptured image obtained in step S1. Note that the brightness of thecaptured image has been calculated in step S32 in order to calculate thehand distance (the position with respect to the z-axis direction), thecontrol section 14 does not need to calculate the brightness again instep S33.

In the fourth embodiment, even if the hand distance is calculated instep S32 and the hand distance is used in the game process (step S37),the portable device 1 can detect a touch operation. That is, theportable device 1 can calculate the hand distance as a game input andstill detect a touch operation. For example, when the brightness of thecaptured image is less than or equal to a predetermined threshold value,the portable device 1 may use the hand distance as a game input, whereaswhen the brightness of the captured image is larger than thepredetermined threshold value, the portable device 1 may determine thata touch operation has been performed.

In step S34, the control section 14 determines whether or not the gameis being played (i.e., during a gameplay). If the determination resultof step S34 is negative, the process of step S35 is performed. On theother hand, if the determination result of step S34 is affirmative, theprocess of step S37 to be described later is performed.

In step S35, the control section 14 determines whether or not a touchoperation has been performed. This determination is made by the methoddescribed in “(Method for detecting touch operation)” above. If thedetermination result of step S35 is affirmative, the process of step S36is performed. On the other hand, if the determination result of step S35is negative, the process of step S31 is performed again. In such a case,the start of the game is on standby until a touch operation is detected.

In step S36, the control section 14 starts the game. After step S36, theprocess of step S31 is performed again. That is, after the process ofstep S36, the determination result of step S34 will be affirmative, anda game process (step S37) in accordance with the operation performed bythe hand of the user will be performed.

On the other hand, in step S37, the control section 14 performs aninformation process (game process) in accordance with the movement ofthe hand of the user. The control section 14 generates a game imagerepresenting the results of the information process, and displays thegame image on the display 2. While the information process may be of anycontent, the control section 14 in the fourth embodiment performs aninformation process of moving a player character in a racing game inaccordance with the hand position.

In step S38, the control section 14 determines whether or not the gameis over. The specific determination method of step S38 may be anymethod. For example, the control section 14 determines whether or not apredetermined game condition (the player character hitting an obstacleor falling off a cliff) has been satisfied. If the determination resultof step S38 is affirmative, the process of step S39 is performed. On theother hand, if the determination result of step S38 is negative, theprocess of step S31 is performed again. Thus, the game progresses as theseries of processes of steps S31 to S34, S37 and S38 are performedrepeatedly until the game is over.

In step S39, the control section 14 determines whether or not to end thegame. The specific method of the determination in step S29 may be anymethod. In the fourth embodiment, it is determined that the game is tobe ended when the user gives a predetermined end instruction (e.g., aninstruction to end the game application). If the determination result ofstep S39 is affirmative, the control section 14 ends the process shownin FIG. 33. On the other hand, if the determination result of step S39is negative, the control section 14 performs the process of step S31again. Thereafter, the series of processes of steps S31 to S39 areperformed repeatedly until it is determined in step S39 to end theprocess.

Note that in the fourth embodiment, the process of determining whetheror not to end the game (step S39) is performed only when the game isover. In other embodiments, the process of determining whether or not toend the game may be performed at any point during the series ofprocesses shown in FIG. 33. For example, the determination process maybe performed when the game is paused.

With the process shown in FIG. 33, the game is started in response to atouch operation by the user (step S36), and the game process, in whichthe movement of the hand of the user is used as a game input, isperformed thereafter until the game is over (steps S31 to S34 and S37).Then, after the game is over, the game is resumed in response todetecting a touch operation again (step S36).

[3. Function/Effect of Fourth Embodiment]

As described above, in the fourth embodiment, the portable device 1detects an operation (touch operation) of bringing an object (the handof the user) into contact with the infrared camera 4 based on thecaptured image (step S35), and performs a predetermined informationprocess based on the operation detection result (step S36). Then, theuser can give an instruction to the portable device 1 by performing anoperation of touching the infrared camera 4. That is, the fourthembodiment enables an unconventional operation, and it is possible toimprove the controllability of the portable device 1.

In the fourth embodiment, the infrared camera 4 is provided on thehousing 10 of the portable device 1 so as to capture an image of (thespace in) the side surface direction of the housing 10. Therefore, theuser can perform an operation of touching the image-capturing devicefrom the side surface direction of the housing 10. Then, the user canhold the portable device 1 with one hand and make a touch operationusing the other hand, thereby providing a portable device with a goodcontrollability.

In the fourth embodiment, the portable device 1 starts performing thepredetermined information process in response to detecting the touchoperation. Herein, “starting performing a predetermined informationprocess” means to include resuming to perform an information processthat has been stopped (or paused). Note that in other embodiments, thepredetermined information process may be stopped (or paused) in responseto detecting a touch operation. As described above, the user caninstruct to start and/or stop an information process by a touchoperation.

In the description above, the portable device 1 calculates objectinformation representing the position of the object based on thecaptured image (step S32), and performs, as the predeterminedinformation process described above, an information process based on theobject information (step S37). Note that in other embodiments, theobject information is not limited to the position of the object, but maybe information representing the shape of the object and/or the movementof the object. As described above, the user can perform a touchoperation and an operation of moving the object, which is a hand, etc.,and it is possible to switch between these two operations. Thus, it ispossible to improve the controllability of the two operations.

In the fourth embodiment, the portable device 1 calculates objectinformation for an object (a hand) in a state where the object is not incontact with the infrared camera 4 or the vicinity thereof, based on thecaptured image (step S32). Then, the portable device 1 can easily switchbetween the process of detecting an operation, and the process ofcalculating the object information.

In the fourth embodiment, the portable device 1 detects a touchoperation based on information (the average brightness value) regardingthe brightness of pixels within a predetermined area of the capturedimage (FIG. 31). Note that while the “predetermined area” is the entirearea of the captured image in the fourth embodiment, it may be, in otherembodiments, any area that is set independently of the image of theobject. As described above, the portable device 1 can easily detect theoperation by using information regarding brightness.

[4. Variation]

Note that in other embodiments, the portable device 1 may calculate, inaddition to the touch operation, the speed of the touch operation and/orthe position of the touch operation, and use them as inputs. An examplewhere the speed of the touch operation or the position of the touchoperation is calculated will be described as a variation of the fourthembodiment.

The portable device 1 may calculate the speed of a touch operation (thespeed at which the hand comes into contact with the infrared camera 4).That is, the portable device 1 may calculate the amount of time takenfrom a state where the brightness of the captured image is relativelylow to a state where it is relatively high (where a touch operation hasbeen performed) so as to calculate the speed of the touch operationbased on the amount of time. Specifically, the portable device 1calculates the amount of time taken from a state where the brightness ofthe captured image is less than or equal to a first threshold value to astate where the brightness is greater than a second threshold value.Note that the second threshold value is greater than the first thresholdvalue, and is a threshold value used for determining whether or not atouch operation has been performed.

As described above, the portable device 1 may calculate the speed of atouch operation based on captured images obtained before the touchoperation is detected. Then, the portable device 1 may change thecontent of process of the predetermined information process inaccordance with the calculated speed. The predetermined informationprocess may be any information process. For example, the portable device1 may perform a first process (e.g., a process of starting the gameprocess), as the predetermined information process, when the speed isgreater than a predetermined value, and perform a second process (e.g.,a process of making a player character perform a predetermined action)different from the first process, as the predetermined informationprocess, when the speed is less than or equal to the predeterminedvalue.

The portable device 1 may calculate the position of the touch operation(the position at which the hand contacts the infrared camera 4). Forexample, when the hand of the user contacts the infrared camera 4, theportable device 1 may determine whether the hand has touched anupper-side portion, a lower-side portion or a central portion of theinfrared camera 4.

Specifically, the portable device 1 may determine the position of thetouch operation by using the hand position in the captured imageobtained before the touch operation is performed and/or the handposition in the captured image obtained when the touch operation isperformed. The portable device 1 calculates the hand position in acaptured image that is obtained immediately before the touch operationis performed (e.g., the captured image shown in (b) of FIG. 31). Thehand position may be the center-of-gravity position calculated in thefirst embodiment. The portable device 1 determines the position of thetouch operation based on the calculated hand position (center-of-gravityposition). For example, the portable device 1 determines that the userhas touched an upper-side portion if the center-of-gravity position islocated above the center of the captured image by a predetermineddistance or more, determines that the user has touched a lower-sideportion if the center-of-gravity position is located below the center ofthe captured image by the predetermined distance or more, and determinesthat the user has touched a central portion if the center-of-gravityposition is within the predetermined distance from the center of thecaptured image. Note that in other embodiments, the portable device 1may calculate the position of the touch operation as coordinates.

Note that in other embodiments, the position of the touch operation maybe determined based on a plurality of captured images that have beenobtained before the touch operation is performed. For example, theportable device 1 may calculate the center-of-gravity position for aplurality of captured images, and determine the position of the touchoperation based on the change in the center-of-gravity position.

As described above, the portable device 1 may determine the position atwhich a touch operation has been performed on the infrared camera 4 (theposition of the touch operation), based on the captured image obtainedwhen the touch operation is detected and/or captured images that havebeen obtained before the detection. Then, the portable device 1 maychange the content of process of the predetermined information processin accordance with the determined position. The predeterminedinformation process may be any information process. For example, theportable device 1 may perform a first process (e.g., a process ofstarting the game process), as the predetermined information process,when the position of the touch operation is on the lower side withrespect to the center of the infrared camera 4, and perform a secondprocess (e.g., a process of making a player character perform apredetermined action) different from the first process, as thepredetermined information process, when the position of the touchoperation is on the upper side with respect to the center of theinfrared camera 4.

<Variation>

(Variation Regarding Portable Device)

While the first to fourth embodiments have been directed to cases wherean information processing device including an image-capturing device(the infrared camera 4) is a portable device, an information processingdevice including an image-capturing device may be of any type. Forexample, in other embodiments, a controller device to be held in a handof a user for performing operations may include an image-capturingdevice. Then, the device for obtaining captured images and performinginformation processes based on the captured images may be a controllerdevice or another information processing device capable of communicatingwith a controller device. For example, a controller for making inputs toa home console-type game device may include an image-capturing device,and the game device may perform an information process based on thecaptured images obtained from the controller. Then, the controller maycalculate information regarding the position, etc. (e.g., position,direction, shape and/or changes thereof) of the hand based on thecaptured images to transmit the information to the game device, and thegame device may perform an information process (game process) based onthe information received from the controller. For example, aninformation processing system including a controller and a main unit,to/from which the controller can be attached/detached, may be used toperform a process in accordance with at least one of the first to fourthembodiments above. For example, the controller may include theimage-capturing device described above, and may calculate informationregarding the position of the hand, etc., (e.g., the positioninformation described above) based on a captured image obtained by theimage-capturing device. Then, the main unit may perform a predeterminedinformation process (e.g., a game process) based on the information.

(Variation Regarding Image-Capturing Device)

In the first to fourth embodiments, the infrared camera 4 is used as anexample of the image-capturing device (i.e., a camera), and the portabledevice 1 obtains information representing the brightness value of theinfrared image (captured images including such information). Now, inother embodiments, the image-capturing device is not limited to theinfrared camera 4, but may be any image-capturing device. For example,in other embodiments, an RGB camera for outputting a captured imagehaving RGB values for each pixel may be used as the image-capturingdevice.

(Variation Regarding Information Calculated Based on Captured Image)

In the first to fourth embodiments, the portable device 1 calculates theposition and/or and the shape of the object (a hand) based on thecaptured image. Now, in other embodiments, the portable device 1 maycalculate the movement of the object as described above.

For example, in other embodiments, the portable device 1 may be ahand-held information processing device, including a housing, and animage-capturing device (the infrared camera 4) capturing an image in theside surface direction of the housing, as in the embodiments describedabove. Thus, the portable device 1 may detect, based on the capturedimage, the rotation and/or the speed of the object included in thecaptured image. The portable device 1 may perform an information processin accordance with the detection result. For example, the portabledevice 1 may detect the rotating action (see FIG. 13) described above,as the rotation of the object, or may detect the speed of the fanningaction (see FIG. 12) or the speed of the touch operation (see FIG. 30)as the speed of the object. Then, the portable device 1 may perform aprocess of rotating the object in accordance with the detected rotation.The portable device 1 may perform a process of moving the object at aspeed in accordance with the detected speed.

In other embodiments, the portable device 1 may detect an operationbased on a combination of the position and the shape of the object. FIG.34 shows an example of an operation using a hand. In FIG. 34, the userperforms an operation of moving the hand toward the infrared camera 4while opening the hand, starting from the closed-hand position. Theportable device 1 may detect such an operation to perform apredetermined information process in response to the operation. Forexample, in FIG. 34, the portable device 1 may generate and display suchan image (movie) that a predetermined object gradually appears from theright end of the display. Then, it is possible to make the user feel asif the object appeared onto the display 2 in accordance with themovement of the hand, thus providing an intuitive and easy-to-understandoperation.

When detecting the operation described above, the portable device 1first calculates the shape and the position of the object included inthe captured image based on the captured image (specifically, theposition of the object with respect to the image-capturing direction).Then, the portable device 1 detects the operation described above basedon a combination of the calculated shape and the calculated position ofthe object. Specifically, the portable device 1 determines whether ornot the operation described above is performed based on whether or notthe hand position is moving toward the infrared camera 4 with the shapeof the hand changing from the first shape (closed shape) into the secondshape (open shape). Note that the method for calculating the handposition and the method for specifying the shape of the hand may bethose described above in the first or second embodiment. Whether or notthe hand position is moving toward the infrared camera 4 may bedetermined based on, for example, whether or not the hand position ismoving in the negative z-axis direction by a predetermined distance ormore (and/or at a predetermined speed or more).

In response to detecting the operation described above, the portabledevice 1 performs a predetermined information process. Specifically, theportable device 1 generates an image (movie) in which the object appearsfrom the right end of the display 2 (the end portion on the side theinfrared camera 4 is provided) and moves toward the center of thescreen, and displays the image (movie) on the display 2. The operationdescribed above can be said to be an operation that makes the user feelas if the user were putting something into the portable device 1.Therefore, by performing a process in which the object is displayed onthe screen in response to the operation described above, it is possibleto provide an intuitive and easy-to-understand operation for the user.Note that the information process to be performed in response to theoperation described above may be any process, and may be, for example, aprocess of installing an application on the portable device 1, a processof putting a data file into a folder, or a process of saving data on theportable device 1. Even when these processes are performed, it ispossible to provide an intuitive and easy-to-understand operation forthe user.

The portable device 1 may detect an opposite operation from theoperation described above, i.e., an operation in which the user movesthe hand away from the infrared camera 4 while closing the hand,starting from the open-hand position. Specifically, the portable device1 determines whether or not the operation described above is performedbased on whether or not the hand position is moving away from theinfrared camera 4 with the shape of the hand changing from the secondshape (open shape) into the first shape (closed shape). In response todetecting this operation, the portable device 1 performs an informationprocess different from the predetermined information process. Forexample, the portable device 1 may generate an image (movie) in whichthe object displayed on the display 2 is moved toward the right end ofthe display 2 (the end portion on the side the infrared camera 4 isprovided) and disappears from the screen at the right end, and displaysthe image (movie) on the display 2. The operation described above can besaid to be an operation that makes the user feel as if the user weretaking (pulling) something out of the portable device 1. Therefore, byperforming a process in which the object disappears from the screen inresponse to the operation described above, it is possible to provide anintuitive and easy-to-understand operation for the user. Note that theinformation process to be performed in response to the operationdescribed above may be any process, and may be, for example, a processof deleting a file, a process of grabbing an object (enabling the userto drag the object), and the like. Even when these processes areperformed, it is possible to provide an intuitive and easy-to-understandoperation for the user.

(Variation Regarding Calculation of Hand Position)

Where the hand position is calculated from the captured image by themethod of the first or second embodiment, the calculated hand positionmay change, in response to a change in the shape or the direction of thehand of the user, even though the hand position has not actually movedsubstantially. Therefore, the portable device 1 may determine a processof calculating the hand position from the captured image based on theshape of the hand and/or the direction of the hand. For example, wherethe hand position is determined from the center-of-gravity position asin the first embodiment, the portable device 1 may vary the process(calculation method) for determining the hand position from thecenter-of-gravity position depending on whether the shape of the hand isa closed shape or an open shape.

(Variation Regarding Input)

In the first to fourth embodiments, the portable device 1 performs apredetermined information process (step S5, S16, S17, S23, S37, etc.) inaccordance with the input (the position of the hand of the user, etc.)detected based on the captured image captured by the infrared camera 4.Now, in other embodiments, the portable device 1 may perform aninformation process based on either this input or an input that is madeon another input section. The other input section is, for example, theinput button 6 and/or the touch panel 3. Thus, it is possible to providea variety of operation methods for the user. For example, in the fourthembodiment, the user can touch the side surface (the position of theinfrared camera 4) of the portable device 1 and can also touch thedisplay 2, thus realizing a novel operation.

As described above, the embodiment described above is applicable as aninformation processing device such as a portable device, for example,with the aim of realizing various inputs using a captured image.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. An information processing system comprising acamera, wherein one or more processor of the information processingsystem is configured to provide execution including: calculating, basedon a camera image obtained by the camera, a position or a direction ofan object included in the camera image; instructing an output device toproduce an output in accordance with the position or the direction ofthe object; and displaying, on a display device, a guide image includinga range image representing a range and an indicator image whose positionor direction changes in accordance with a change in the output withinthe range, wherein the processor is configured to display, on thedisplay device, the guide image including a predetermined devicereference image so that a positional relationship between the indicatorimage and the predetermined device reference image corresponds to apositional relationship between the object and the camera.
 2. Theinformation processing system according to claim 1, wherein the camerais configured to capture an image in a side surface direction of ahousing of the information processing system.
 3. The informationprocessing system according to claim 1, wherein the indicator image isan image representing the object.
 4. The information processing systemaccording to claim 1, wherein when a position of the object cannot becalculated, the processor changes a display mode of the guide image, ascompared with that when the position of the object is calculated.
 5. Theinformation processing system according to claim 1, wherein when thecalculated position of the object is further away toward anun-calculatable area from a predetermined reference position by apredetermined distance or more, the processor changes a display mode ofthe guide image, as compared with that when the calculated position iswithin the predetermined distance.
 6. The information processing systemaccording to claim 1, wherein the processor is configured to calculate,based on the camera image, a position of the object with respect to animage-capturing direction of the camera.
 7. The information processingsystem according to claim 1, wherein: the processor is configured tocontrol an output to the output device so that the output is inaccordance with a relationship between the calculated position of theobject and a predetermined reference position; and the processor isconfigured to display, on the display device, the guide image includinga process reference image placed at a position corresponding to thereference position within the range represented by the range image.
 8. Anon-transitory computer-readable storage medium storing an informationprocessing program to be performed by a computer of an informationprocessing system comprising a camera, the information processingprogram instructing the computer to execute: calculating, based on acamera image obtained by the camera, a position or a direction of anobject included in the camera image; instructing an output device toproduce an output in accordance with the position or the direction ofthe object; and displaying, on a display device, a guide image includinga range image representing a range and an indicator image whose positionor direction changes in accordance with a change in the output withinthe range, wherein the guide image includes a predetermined devicereference image so that a positional relationship between the indicatorimage and the predetermined device reference image corresponds to apositional relationship between the object and the camera.
 9. Aninformation processing device comprising: a camera, and one or moreprocessors at least configured to: calculate, based on a camera imageobtained by the camera, a position or a direction of an object includedin the camera image; instruct an output device to produce an output inaccordance with the position or the direction of the object; anddisplay, on a display device, a guide image including a range imagerepresenting a range and an indicator image whose position or directionchanges in accordance with a change in the output within the range,wherein the guide image includes a predetermined device reference imageso that a positional relationship between the indicator image and thepredetermined device reference image corresponds to a positionalrelationship between the object and the camera.
 10. An informationprocessing method to be performed on an information processing systemcomprising a camera, the method comprising: calculating, based on acamera image obtained by the camera, a position or a direction of anobject included in the camera image; instructing an output device toproduce an output in accordance with the position or the direction ofthe object; and displaying, on a display device, a guide image includinga range image representing a range and an indicator image whose positionor direction changes in accordance with a change in the output withinrange; wherein the guide image includes a predetermined device referenceimage so that a positional relationship between the indicator image andthe predetermined device reference image corresponds to a positionalrelationship between the object and the camera.
 11. The informationprocessing system according to claim 1, wherein the predetermined devicereference image is a camera reference image so that a positionalrelationship between the indicator image and the camera reference imagecorresponds to the positional relationship between the object and thecamera.
 12. The non-transitory computer-readable storage mediumaccording to claim 8, wherein the predetermined device reference imageis a camera reference image so that a positional relationship betweenthe indicator image and the camera reference image corresponds to thepositional relationship between the object and the camera.
 13. Theinformation processing device according to claim 9, wherein thepredetermined device reference image is a camera reference image so thata positional relationship between the indicator image and the camerareference image corresponds to the positional relationship between theobject and the camera.
 14. The method according to claim 10, wherein thepredetermined device reference image is a camera reference image so thatthe positional relationship between the indicator image and the camerareference image corresponds to a positional relationship between theobject and the camera.